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EXECUTIVE  SUMMARY 


The  U.S.  Navy  has  embarked  on  a  new  computing  resources  standardization 
effort,  the  Next  Generation  Computer  Resources  (NGCR)  program.  The  program  is 
designed  to  fulfill  the  Navy's  need  for  standard  computing  resources  while 
allowing  it  to  take  advantage  of  commercial  products  and  investments  and  to 
field  technological  advances  more  quickly.  The  program  has  identified  ten 
(10)  interface  areas  for  standardization.  The  Operating  Systems  Standards 
Working  Group  (OSSWG)  has  been  tasked  with  the  identifying  interfaces  for 
operating  systems  (OS) .  The  general  requirements  for  the  OS  standard  include 
Ada,  real-time,  distribution,  multilevel  security,  fault  tolerance,  and  that 
it  be  realizable  on  heterogeneous  systems.  An  initial  OS  interface  standard 
is  expected  in  1993,  and  the  final  standard  is  expected  to  be  usable  in  the 
procurement  of  Navy  systems  in  fiscal  year  1996. 

The  NGCR  approach  is  that  of  an  open  systems  architecture  based  on  the 
establishment  of  interface  standards.  The  application  of  these  standards  will 
change  the  Navy's  approach  from  one  of  buying  standard  computers  to  one  of 
procuring  computer  resources  which  satisfy  the  interfaces  defined  by  the 
standards.  These  standards  will  be  applied  to  procurements  at  the  project 
level  rather  than  a  Navy-wide  procurement  level.  The  interface  standards  will 
be  based,  to  the  greatest  extent  possible,  on  existing  industry  standards.  In 
cases  where  existing  industry  standards  do  not  meet  Navy  mission  critical 
needs,  the  approach  is  to  enhance  the  existing  standards  jointly  with 
industry,  thus  assuring  the  most  widely  accepted  set  of  nonproprietary 
commercially  based  Navy  interface  standards  possible. 

The  OSSWG  met  throughout  1989  and  early  1990  to  define  requirements,  to 
identify  candidates  for  the  OS  interface,  and  to  define  an  evaluation  process 
(see  Evaluation  Process  Report).  Seventeen  classes  of  requirements  have  been 
identified.  One  class  is  programmatic  issues;  the  other  sixteen  are  technical 
service  classes.  Within  each  service  class  are  requirements  which  define  the 
evaluation  criteria.  The  technical  evaluation  criteria  are  documented  in  the 
OSSWG  Requirements  Document. 

Seven  OS  interfaces  were  identified  as  candidates: 


1. 

Alpha 

2. 

ARTX 

3. 

Cronus/SDOS 

4. 

iRMX 

5. 

Mach/RT - Mach/T - Mach 

6. 

ORKID 

7. 

POSIX 

i 
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Eight  representative  application  domains  were  described.  Weights, 
called  Weight  Set  2,  were  assigned  to  each  of  the  OSSWG  Requirements  Service 
Classes  for  these  domains.  Other  weights,  called  Weight  Set  1,  were  also 
assigned  to  each  criterion  within  a  service  class  to  determine  the  relative 
importance  of  the  requirement  to  the  service  class. 

Evaluators  from  the  Navy,  other  government  agencies,  industry,  and 
academia  participated  in  rating  the  capabilities  of  each  of  the  OS  interfaces 
against  required  capabilities  defined  in  the  OSSWG  Requirements  Document. 

These  evaluations  produced  the  raw  scores  (0-10,  0  lowest)  that  were  tabulated 
with  the  two  weight  sets  applied. 

Preliminary  results  were  presented  to  the  OSSWG  at  the  meeting  of  6-8 
March  1990.  The  candidate  OS  interfaces  were  not  identified  by  name,  but 
rather  by  days  of  the  week.  As  the  results  were  preliminary,  this  was  done  to 
avoid  any  undue  influence  on  evaluations  that  were  still  being  compiled. 

Final  results  were  presented  to  the  OSSWG  at  the  meeting  of  17-19  April 
1990.  During  this  meeting  all  candidates  were  identified  by  name. 

This  document  includes  the  results  of  the  evaluation  and  the  results  of 
the  analysis  performed  on  the  evaluation's  raw  scores.  For  conclusions  and/or 
discussions  of  the  results,  see  the  Recommendation  Report. 
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FOREWORD 


The  work  reported  here  was  conducted  over  a  period  of  a  little  more  than 
one  year  by  a  joint  team  of  Navy,  other  government,  industry,  and  academic 
experts  in  the  field  of  computer  operating  systems.  Only  a  few  of  the  Navy 
participants  were  actually  funded  to  directly  participate  in  this  process. 

The  report  was  funded  under  NUSC  Job  Order  Number  A45146,  Next 
Generation  Computer  Resources.  The  sponsoring  activity  is  Space  and  Naval 
Warfare  Systems  Command,  through  the  work  of  the  Operating  Systems  Standards 
Working  Group  (OSSWG) .  The  OSSWG  management  structure  is  as  follows: 

NGCR  Program  Manager,  Mr.  H.  Mendenhall,  SPAWAR  324 
NGCR  OSSWG  Co-Chairman,  CDR  R.  Barbour,  SPAWAR  324 
NGCR  OSSWG  Co-Chairman,  Ms.  T.  Oberndorf,  NADC 
Approach  Subgroup  Chairman,  Mr.  T.  Conrad,  NUSC 
Requirements  Subgroup  Chairman,  Mr.  R.  Bergman,  NOSC 
Available  Technology  Subgroup  Chairman,  Mr.  J.  Oblinger,  NUSC 

Although  the  report  is  the  result  of  work  performed  by  the  entire 
membership  of  the  OSSWG,  the  following  OSSWG  members  actively  performed  the 
evaluation  of  the  final  seven  candidates: 


CDR  Richard  Barbour 
Richard  Bergman 
Paul  Bickness 
Richard  Brogan 
Dale  Brouhard 
Gregory  Bussiere 
Antonio  Carangello 
Gordon  Caswell 
Thomas  Conrad 
B.  Dasarathy 
Larry  Dauber t 
Isobel  Davis 
Steven  Davis 
Dr.  Thomas  Drake 
Richard  Dvorchak 
LT  Karl  Fairbanks 
Gary  Fisher 
Lester  Fraim 
Dr.  Karen  Gordon 
Dr.  Mars  Gralia 
Daniel  Green 
Raymond  Gretlein 


SPAWAR  324 

NOSC 

Mitre 

Booz,  Allen,  &  Hamilton 

NOSC 

NUSC 

Mitre 

ESL 

NUSC 

Concurrent  Computer 
Rockwell  International 
Raytheon 
DGM&S 

Clemson  University 

Intel 

NWC 

NIST 

Honeywell 

IDA 

JHU/APL 

NAVSWC 

Dynamics  Research 
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Joseph  Gwinn 

Raytheon 

Barbara  Haleen 

Unisys 

James  Hall 

NIST 

Neil  Henderson 

Litton  Data  Systems 

Gail  Holmes 

NUSC 

Steven  Howell 

NAVSWC 

John  Johnson 

NAC 

Daniel  Juttelstad 

NUSC 

Kari  Kruempel 

Unisys 

Dr.  James  Leathrum 

Clemson  University 

Warren  Loper 

Texas  Instruments 

Dr.  Douglass  Locke 

IBM 

Warren  Loper 

NOSC 

Michael  Morgan 

Pacific  International  Center  for 
High  Technology  Research  (PICHTR) 

Dr.  John  F.  Nixon 

General  Electric  Co.  Advanced 
Technology  Loaboratories 

Patricia  Oberndorf 

NADC 

James  Oblinger 

NUSC 

Frank  Prindle 

NADC 

John  Reed 

DEC 

Carl  Reinert 

Computer  Based  Systems 

Helmut  Roth 

NAVSWC 

Dr.  Timothy  Saponas 

Intel 

John  Shea 

NOS 

Del  Swanson 

Unisys 

Maria  Voreh 

NRL 

Patrick  Watson 

IBM 

We  would  like  to  thank  Carl  Schraiedekamp  (NADC)  for  his  help  with  the 
data  collection  and  analysis. 

We  would  especially  like  to  take  this  opportunity  to  thank  the  United 
States  industry  and  academia  for  the  staunch  support  of  and  participation  in 
this  working  group  and  would  like  to  strongly  encourage  their  continued 
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CHAPTER  1 
INTRODUCTION 


The  U.S.  Navy  has  embarked  on  a  new  computing  resources  standardization 
effort,  the  Next  Generation  Computer  Resources  (NGCR)  program.  The  program  is 
designed  to  fulfill  the  Navy's  need  for  standard  computing  resources  while 
allowing  it  to  take  advantage  of  commercial  products  and  investments  and  to 
field  technological  advances  more  quickly.  The  program  has  identified  ten  ‘ 
Jn— (iO)  interface  areas  for  standardization.  The  Operating  Systems  Standards 
Working  Group  '^©frSWS^'has  been  tasked  with  identifying  a  set  of  interfaces  for 
operating  systems  (OS).  The  general  requirements  for  the  OS  standard  include 
Ada,  real-time,  distribution,  multilevel  security,  fault  tolerance,  and  that 
it  be  realizable  on  heterogeneous  systems.  An  initial  OS  interface  standard 
is  expected  in  1993,  and  the  final  standard  is  expected  to  be  usable  in  the 
procurement  of  Navy  systems  in  fiscal  year  1996^^ 


1.1  OVERVIEW 


This  report  summarizes  the  results  of  the  Next  Generation  Computer 
Resources  (NGCR)  Operating  Systems  Standards  Working  Group  (OSSWG)  evaluation 
of  candidates  for  the  Operating  System  Interface  (OSIF)  Baseline.  Extensive 
details  of  the  preplanning  for  the  evaluation  are  provided  in  the  Evaluation 
Process  Report.  The  recommendation  for  the  OSIF  baseline  is  described  in  the 
Recommendation  Report.  An  After  Action  Report  provides  general  recommendations 
and  lessons  learned.  ^  ^ 


<7 


The  OSSWG  effort  began  early  in  1989  with  the  first  open  meeting  in 
March.  Participation  by  industry,  academia,  and  government  was  solicited. 
The  Navy  goal  was  to  get  a  qualified  set  of  people  to  examine  a  manageable 
number  of  candidate  interface  sets  against  Navy  requirements.  Navy 
requirements,  candidate  interfaces,  and  a  selection  process  had  to  be 
developed  and  identified. 


Meeting  attendance  was  open.  However,  evaluators  were  required  to 
qualify  by  attendance  at  two  or  more  meetings.  The  set  of  evaluators 
submitting  evaluations  consisted  of  21  Navy  and  27  non-Navy  representatives. 
Government  (other  than  Navy),  academia,  and  industry  constituted  the  non-Navy 
participation. 


To  accomplish  the  work,  three  subgroups  were  established:  Available 
Technology,  Requirements,  and  Approach. 
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1.1.1  Available  Technology  Subgroup 

The  Available  Technology  Subgroup  identified  approximately  110 
candidates  for  evaluation.  By  reviewing  documentation  of  these  candidates, 
the  number  was  reduced  to  10  candidates  (See  Appendix  A.  Description  of 
Candidates) : 


1 .  Alpha 

2.  ARTX 

3 .  Cronus 

4.  iRMX 

5 .  Mach 

6 .  ORKID 

7.  POSIX 

8.  MTOS 

9 .  SDOS 

10.  T-Mach 

The  Available  Technology  Subgroup  arranged  to  have  representatives 
present  information  about  their  operating  system  interface  to  the  OSSWG  in 
December  1989  and  January  1990.  Mach,  Trusted  Mach  and  RT  Mach  were  combined 
as  were  Cronus  and  SDOS,  bringing  the  number  of  candidates  to  eight.  After  the 
December  1989  meeting,  Industrial  Programming  Incorporated  (IPI)  withdrew  MTOS 
from  consideration. 

1.1.2  Requirements  Subgroup 

The  Requirements  Subgroup  identified  16  classes  of  requirements.  These 
sixteen  service  classes  all  addressed  technical  areas.  Each  service  class 
contained  requirements  which  defined  the  evaluation  criteria. 

1.1.3  Approach  Subgroup 

The  Approach  Subgroup  defined  the  evaluation  process,  including  a 
seventeenth  service  class  (service  class  0)  containing  programmatic  evaluation 
criteria.  (See  Evaluation  Process  Report).  The  Approach  Subgroup  was  also 
responsible  for  the  definition  of  the  representative  application  domains 
(RADs) .  A  set  of  eight  RADs  was  defined. 
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1.2  ORGANIZATION 

This  document  presents  the  results  of  the  OSSWG  evaluation  process. 
Chapter  2  addresses  the  work  of  the  Available  Technology  Subgroup.  Weight 
sets  developed  by  the  Approach  Subgroup  and  the  entire  OSSWG  are  discussed  in 
Chapter  3.  A  description  of  the  effort  expended  by  evaluators  is  prc-^nted  in 
Chapter  4.  Chapter  5  presents  a  description  of  the  data  acquisition  process 
including  information  about  the  briefings  on  the  OS  interfaces  that  were  given 
to  the  OSSWG.  Chapter  6  provides  a  summary  of  the  results.  Since  there  was 
no  one  candidate  that  was  clearly  superior  to  all  others,  the  OSSWG  considered 
alternate  views  of  the  data.  These  are  discussed  in  Chapter  7.  Chapter  S 
concludes  this  report. 
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CHAPTER  2 

SCREENING  PROCESS  RESULTS 


From  the  initial  Next  Generation  Computer  Resources  OSSWG  Brief  to 
Industry  in  March  1989,  the  Available  Technology  (AT)  Subgroup  was  tasked  with 
collecting  information  on  available  operating  systems  interfaces.  A  list  of 
existing  operating  systems  interfaces  that  the  subgroup  felt  might  fulfill  any 
of  the  requirements  of  NGCR  was  compiled.  This  list  included  110  operating 
systems,  research  projects  in  OS  technology,  and  OS  standards  activities.  A 
brief  investigation  of  each  of  these  systems  was  made,  and  a  more  detailed 
survey  of  the  most  promising  was  compiled  and  included  in  the  Available 
Technology  Report. 

At  the  conclusion  of  this  collection  process,  the  Available  Technology 
Subgroup  had,  as  its  major  agenda  item  in  October  1989,  the  narrowing  of  the 
operating  systems  interfaces  candidate  list.  This  "early  screening,"  as 
specified  by  the  Approach  Subgroup  of  the  NGCR  OSSWG,  was  performed  at  the 
OSSWG  Meeting  held  14-16  October  1989. 

To  accomplish  this  early  screening,  a  set  of  criteria  was  defined  with 
which  to  narrow  the  complete  Available  Technology  list  of  operating  systems 
interfaces  and  interface  standards  to  a  number  manageable  for  the  formal 
evaluation  process.  The  stated  purpose  of  this  process  was  to  narrow  the 
complete  list  to  only  those  interfaces  which  had  the  potential  of  fulfilling 
the  NGCR  OSSWG  requirements.  Interfaces  that  were  "too  close  to  call"  could 
be  expected  to  be  on  the  candidate  list.  The  formal  evaluation  process  would 
provide  the  final  OSSWG  selection. 

Various  methods  for  narrowing  the  rather  large  list  of  operating  system 
interface  candidates  were  discussed  and  exercised  over  several  months.  The 
appropriate  method  had  a  minimum  number  of  criteria,  yet  provided  a  fair  and 
equitable  separation  of  the  interfaces  deemed  to  be  valuable  to  the  OSSWG  from 
those  which  did  not. 

The  screening,  as  finally  accomplished,  consisted  of  two  separate, 
complementary  sets  of  early  screening  criteria.  The  first  method  was  known  as 
the  Decision  Option  Paper  Method  (DOP  Method)  and  was  based  on  a  comparison  of 
operating  system  capabilities  against  the  DOP  Technology  area  requirements 
described  in  the  NGCR  DOP  document.  The  second  method  was  called  the  Positive 
Negative  Method  (PN  Method)  and  was  based  on  a  specific  set  of  criteria  for 
rating  candidates.  Positive  criteria  included  such  things  as  whether  it  was  a 
current  interface  standard;  negative  criteria  included  such  things  as  whether 
it  was  simply  a  narrowly  focused  research  tool.  (See  the  Available  Technology 
Screening  Report  for  information  on  the  exact  process  and  criteria.) 
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Using  these  screening  criteria,  the  Available  Technology  Subgroup 
reduced  the  number  of  candidates  to  ten.  The  names  of  these  ten  were:  ALPHA, 
ARTX,  CRONUS,  iRMX,  MACH,  MTOS,  ORKID,  POSIX,  TMACH,  and  SDOS. 

As  OSSWG  approached  the  22-26  January  meeting,  there  was  some 
consolidation  in  the  candidate  list.  Previously,  in  early  December,  the  San 
Diego  meeting  had  produced  ten  finalists  and  the  realization  that  several  of 
the  candidates  emanated  from  the  same  root  and  should  therefore  be  presented 
as  one. 

After  some  discussion  with  both  BBN,  representing  the  CRONUS  candidate, 
and  Odyssey  Research  Institute,  representing  the  SDOS  candidate,  it  was  agreed 
that  they  should  be  presented  as  one  candidate.  The  funding  for  both  of  these 
products  originated  from  Rome  Air  Development  Center  (RADC) .  SDOS  was  a 
product,  being  created  by  OSI,  to  build  secure  services  into  the  CRONUS 
products.  As  one  candidate,  CRONUS/SDOS  would  be  evaluated  as  a  candidate 
with  a  greater  breadth  of  capability  than  either  alone.  And  most  importantly, 
they  were  intended  to  work  together:  this  was  not  a  contrived  pairing. 

In  a  similar  manner,  both  Mach  and  Trusted  Mach  were  on  the  list  of 
candidates.  It  seemed  natural  to  group  the  Mach  family  of  systems  together. 
These  included  Mach  represented  by  Carnegie  Mellon  University  (CMU) ,  Trusted 
Mach  represented  by  Trusted  Information  Systems,  and  Real-Time  Mach  (RT  Mach) 
represented  by  CMU. 

An  additional  change  to  the  candidate  list  occurred  when  MTOS, 
represented  by  the  manufacturer,  Industrial  Programming  Inc.  (IPI) ,  decided  to 
withdraw  from  the  evaluation  process.  The  reasons  cited  by  IPI  were  concerns 
about  copyrights  and  releasing  proprietary  information.  These  areas  of 
concern  have  always  been  topics  of  discussion  in  OSSWG;  and  certainly,  the 
winner  of  the  evaluation  would  have  to  be  prepared  to  provide  full  access  to 
the  interface  documentation. 

The  candidates  selected  by  this  process  proved  to  be  very  capable.  They 
provided  valuable  insight  into  their  philosophies  of  operating  system 
interface  designs.  Noting  the  strength  of  each  of  the  candidates,  the  early 
screening  process  accomplished  its  intended  purpose. 
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CHAPTER  3 
WEIGHT  SETS 


In  order  to  better  understand  how  the  results  of  the  evaluation  relate 
to  the  reality  of  Navy  systems,  various  scores  within  the  evaluation  were 
weighted.  The  weights  allowed  the  data  reduction  techniques  used  in-  the 
evaluation  analysis  to  include  the  relative  importance  of  each  data  point  in 
the  combined  scores. 


3 . 1  OVERVIEW 

There  were  two  sets  cf  weights.  The  weights  applied  in  the 
transformation  from  criteria  scores  to  service  class  scores  were  called  Weight 
Set  1.  This  weight  set  described  the  relative  importance  of  each  criterion 
score  to  its  particular  service  class.  The  weights  applied  in  the 
transformation  of  service  class  scores  to  representative  application  domains 
(RADs)  were  called  Weight  Set  2.  This  weight  set  described  the  relative 
importance  of  each  service  class  to  each  representative  application  domain. 

All  weights  were  on  a  0.0  to  10.0  scale  with  0.0  being  the  lowest  possible 
weight  and  10.0  being  the  highest. 

3.1.1  Weight  Set  1 

The  criteria  within  service  classes  2  through  16  were  weighted  in  order 
to  arrive  at  a  single  service  class  score  (for  each  service  class) .  Table  3- 
1  shows  the  Weight  Set  1.  Included  in  this  table  is  one  standard  deviation 
(sigma)  of  the  weight  as  determined  by  the  standard  mathematical  formula.  In 
the  application  of  these  weights,  each  weight  is  multiplied  by  the  criterion 
score,  then  the  resultant  scores  for  a  service  class'  criteria  are  added 
together  and  normalized  (by  the  total  sum  of  the  weights  in  the  service 
class).  The  normalized  weighted  sum  of  scores  represents  the  service  class 
score . 
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TABLE  3-1.  WEIGHT  SET  1 


Cr. 

Wt. 

Sigma 

Cr. 

Wt. 

Sigma 

Cr. 

Wt. 

Sigma 

Cr. 

Wt. 

Sigma 

2.1 

1.00 

0.00 

6.8 

8.79 

2.15 

9.6 

7.89 

2.82 

13.3 

4.57 

1.81 

3.1 

7.0 

0.0 

6.9 

9.29 

1.71 

9.7 

7.48 

2.15 

13.4 

7.61 

2.60 

3.2 

7.0 

0.0 

6.10 

9.00 

1.91 

9.8 

8.48 

1.99 

13.5 

5.79 

2.87 

3.3 

5.0 

0.0 

6.11 

8.67 

2.20 

9.9 

6.78 

1.99 

13.6 

6.54 

2.60 

3.4 

8.0 

0.0 

6.12 

7.92 

2.45 

9.10 

6.26 

2.21 

13.7 

7.86 

2.26 

3.5 

6.0 

0.0 

6.13 

7.33 

2.32 

9.11 

6.74 

2.19 

13.8 

6.86 

2.21 

3.6 

6.0 

0.0 

6.14 

8.00 

2.48 

9.12 

7.89 

2.38 

13.9 

5.93 

2.71 

3.7 

3.0 

0.0 

6.15 

4.92 

2.86 

9.13 

5.59 

2.04 

13.10 

6.61 

2.17 

3.8 

5.0 

0.0 

6.16 

8.79 

2.15 

9.14 

6.74 

2.55 

13.11 

7.25 

2.19 

3.9 

5.0 

0.0 

6.17 

7.58 

2.83 

10.1 

6.37 

3.50 

14.1 

8.63 

1.74 

3.10 

3.0 

0.0 

6.18 

7.33 

2.76 

10.2 

6.22 

3.14 

14.2 

8.75 

2.07 

3.11 

10.0 

0.0 

6.19 

7.75 

2.51 

11.1 

7.96 

2.66 

14.3 

6.87 

2.65 

3.12 

3.0 

0.0 

6.20 

7.00 

2.41 

11.2 

6.50 

2.96 

15.1 

8.96 

1.72 

3.13 

5.0 

0.0 

7.1 

8.71 

1.81 

11.3 

5.92 

2.78 

15.2 

7.96 

2.08 

3.14 

4.0 

0.0 

7.2 

8.88 

1.90 

11.4 

5.71 

3.03 

15.3 

6.70 

2.13 

3.15 

10.0 

0.0 

7.3 

8.88 

1.90 

11.5 

6.58 

2.86 

15.4 

5.63 

3.13 

3.16 

10.0 

0.0 

7.4 

8.79 

1.84 

11.6 

5.75 

1.73 

15.5 

5.78 

3.13 

3.17 

7.0 

0.0 

7.5 

8.75 

1.92 

11.7 

6.33 

2.57 

15.6 

8.37 

1.90 

3.18 

3.0 

0.0 

7.6 

8.21 

2.19 

11.8 

7.00 

2.17 

15.7 

6.67 

2.76 

3.19 

6.0 

0.0 

7.7 

7.96 

1.94 

11.9 

7.29 

2.39 

16.1 

8.08 

2.70 

3.20 

1.0 

0.0 

7.8 

6.79 

2.81 

11.10 

6.67 

2.85 

16.2 

7.38 

3.20 

3.21 

7.0 

0.0 

7.9 

7.46 

2.15 

11.11 

6.13 

2.72 

16.3 

7.81 

2.62 

3.22 

9.0 

0.0 

7.10 

6.00 

3.01 

11.12 

5.67 

2.32 

16.4 

7.81 

2.62 

3.23 

4.0 

0.0 

7.11 

5.92 

2.95 

11.13 

5.17 

2.28 

16.5 

7.42 

2.79 

3.24 

2.0 

0.0 

8.1 

7.58 

2.81 

11.14 

5.83 

2.37 

16.6 

4.62 

2.47 

4.1 

1.00 

0.00 

8.2 

6.54 

2.15 

12.1 

7.58 

2.75 

16.7 

8.62 

2.40 

5.1 

7.11 

2.28 

8.3 

6.83 

2.44 

12.2 

7.29 

2.51 

16.8 

8.50 

2.50 

5.2 

7.15 

2.20 

8.4 

5.67 

2.66 

12.3 

8.08 

1.72 

16.9 

5.85 

2.80 

5.3 

7.44 

2.53 

8.5 

6.0 

2.55 

12.4 

7.08 

2.28 

16.10 

5.46 

2.83 

5.4 

6.04 

2.55 

8.6 

5.79 

2.81 

12.5 

7.58 

2.48 

16.11 

7.38 

2.93 

5.5 

6.89 

3.26 

8.7 

5.38 

2.73 

12.6 

6.33 

2.97 

16.12 

7.04 

2.95 

5.6 

7.67 

2.63 

8.8 

5.88 

2.56 

12.7 

6.33 

2.84 

16.13 

7.92 

2.65 

6.1 

7.71 

2.31 

8.9 

5.29 

2.99 

12.8 

6.21 

3.13 

16.14 

5.58 

3.09 

6.2 

7.08 

2.83 

8.10 

3.57 

2.15 

12.9 

6.96 

2.93 

16.15 

7.42 

2.94 

6.3 

7.29 

3.04 

9.1 

8.48 

2.24 

12.10 

6.96 

2.93 

16.16 

7.81 

2.43 

6.4 

7.08 

2.39 

9.2 

7.11 

3.17 

12.11 

6.67 

2.70 

16.17 

5.23 

3.37 

6.5 

6.25 

3.31 

9.3 

8.00 

2.66 

12.12 

6.25 

2.49 

16.18 

6.12 

2.88 

6.6 

5.75 

2.89 

9.4 

5.85 

3.18 

13.1 

8.82 

1.59 

6.19 

8.15 

2.49 

6.7 

9.38 

1.58 

9.5 

7.70 

2.84 

13.2 

8.68 

2.07 

6.20 

7.50 

3.62 
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Since  the  criteria  of  service  class  1  (General)  are  not  necessarily 
related  to  each  other,  it  was  determined  that  no  weights  should  be  generated  for 
this  service  class.  Additionally,  service  class  0  criteria  are  related  to 
programmatic  issues;  therefore,  it  was  decided  that  the  NGCR  program  office 
would  be  the  best  organization  to  determine  the  relative  importance  of  criteria 
within  this  class. 

The  weights  of  Weight  Set  1  were  developed  by  attendees  of  the  December 
1989  OSSWG  meeting.  The  meeting  was  divided  arbitrarily  into  two  groups.  Each 
group  generated  a  subset  of  the  weights.  Each  criterion  was  read  aloud  and 
discussed.  Then  each  member  of  the  group  weighted  the  criterion  from  0  to  10  as 
to  its  importance  in  the  service  class. 

The  weights  for  each  criterion  were  arrived  at  by  averaging  all  the 
December  attendee  weight  scores  for  that  criterion.  The  exception  to  the  above 
process  was  for  service  class  3  (Security  and  Capability),  because  the 
requirements  for  this  service  class  were  not  mature  by  the  December  1989 
meeting.  The  weights  for  service  class  3  criteria  were  set  once  the  service 
class  was  complete,  but  before  the  evaluation  began  using  a  consensus  process. 
The  weights  for  this  service  class  were  generated  by  the  developers  of  this 
service  class'  criteria. 

The  individual  weight  scores ,  as  well  as  the  resulting  average ,  were  not 
disclosed  to  the  OSSWG  membership  until  after  the  evaluation  of  the  seven 
candidates  was  complete.  This  was  to  eliminate  the  risk  of  prior  knowledge  of 
the  weights  affecting  the  scoring  results  of  the  evaluators.  All  weights, 
including  service  class  3  weights,  were  fixed  and  concealed  before  the  official 
evaluation  process  began. 

3.1.2  WEIGHT  SET  2 


Weight  Set  2  was  developed  by  the  Approach  Subgroup  of  the  OSSWG  during 
the  OSSWG' s  September  1989  meeting.  These  weights  were  fixed  and  approved  by 
OSSWG  consensus  during  the  October  1989  meeting.  The  weights  were  arrived  at 
through  an  iterative  process.  In  the  September  meeting  the  members  of  the 
Approach  Subgroup  generated  the  service  class  weights  for  each  RAD.  The  weights 
from  each  subgroup  member  were  disclosed  and  a  discussion  of  each  weight  ensued. 
The  members  of  the  Approach  Subgroup  were  given  the  time  between  the  September 
1989  and  October  1989  OSSWG  meetings  to  reconsider  and  resubmit  their  weights. 

At  the  October  1989  OSSWG  meeting,  the  weights  from  each  Approach  Subgroup 
member  for  each  service  class/RAD  pair  were  averaged  to  generate  the  final 
weight  set.  As  with  Weight  Set  1,  Weight  Set  2  only  weighted  service  classes  2 
through  16.  Table  3-2  lists  Weight  Set  2.  Table  3-3  presents  the  results  of  an 
analysis  performed  on  Weight  Set  2  before  the  evaluation  of  the  seven 
candidates . 
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The  total  demand  placed  on  an  Operating  Systems  Interface  (OSIF)  by 
representative  application  domains  is  indicated  by  the  total  weights  in  the  last 
row  of  Table  3-2.  The  most  demanding  RADs  are  Amethyst,  Topaz,  Emerald, 

Diamond,  and  Sapphire. 


TABLE  3-2.  WEIGHT  SET  2 


c 

R 

0 

A 

G 

T 

E 

D 

S 

L 

U 

P 

M 

A 

0 

M 

I 

A 

A 

B 

A 

E 

R 

P 

E 

A 

P 

S 

Y 

L 

T 

N 

A 

R 

M 

P 

S 

H 

E 

Z 

A 

O 

H 

Y 

T 

L 

N 

I 

# 

S 

D 

D 

R 

T 

E 

2 

3.2 

5.3 

4.2 

5.8 

6.0 

7.2 

7.8 

6.0 

3 

5.8 

2.3 

9.0 

3.2 

4.8 

9.3 

4.3 

7.6 

4 

6.4 

5.5 

ra 

ra 

ra 

4.8 

5.5 

6.2 

5 

ra 

ra 

7.5 

7.5 

6.2 

7.2 

8.5 

7.2 

6 

9.2 

0.5 

ra 

ra 

4.2 

ra 

ra 

3.4 

7 

5.0 

5.5 

4.5 

6.7 

6.3 

5.3 

6.5 

6.0 

8 

3.8 

5.2 

9.5 

2.8 

6.2 

5.3 

6.0 

8.8 

9 

4.0 

4.5 

5.5 

5.3 

5.2 

6.3 

8.2 

10 

3.0 

ra 

ra 

ra 

6.2 

7.3 

6.5 

7.6 

11 

3.4 

6.0 

7.5 

8.5 

8.2 

9.2 

8.7 

8.2 

12 

6.0 

3.0 

5.5 

4.0 

4.3 

5.0 

5.3 

6.8 

13 

5.4 

5.5 

7.0 

4.3 

4.8 

5.2 

8.3 

7.8 

14 

4.6 

3.5 

5.0 

4.3 

6.3 

5.2 

5.5 

4.6 

15 

3.8 

7.2 

7.8 

7.0 

6.8 

ra 

8.4 

16 

m 

ra 

6.8 

8.2 

8.8 

8.7 

8.3 

8.8 

Total  Weight 

71.4 

69.5 

96.4 

75.3 

92.0 

93.4 

96.9 

105.6 
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Table  3-3  indicates  the  degree  to  which  the  eight  RADs  differ  in  their 
weights.  The  entries  are  the  sums  of  the  differences  of  the  weights  for  the 
domains  indicated  by  the  column  and  row.  Based  on  this  data,  the  smallest 
distinction  provided  by  these  application  domains  is  between  Emerald  and 
Diamond.  In  that  case  the  difference  between  the  sum  of  the  weights  is  16.9. 
This  still  averages  to  a  difference  greater  than  1  for  each  weight  for  service 
classes  2  through  16.  The  last  column  of  Table  3-3  is  a  row  matrix  summation  of 
the  differences.  This  summation  gives  the  total  weight  difference  between  a  RAD 
and  all  other  RADs.  Other  analyses  of  Weight  Set  2  can  be  found  in  Appendix  B, 
Further  Analysis  of  Weight  Set  2. 


TABLE  3-3.  TOTAL  DIFFERENCE  WEIGHTS  OF  SERVICE  CLASSES  2  THROUGH  16 
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R 
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RUBY 

0.0 

32.3 

36.0 

40.5 

37.2 

42.3 

46.4 

46.2 

280.9 

OPAL 

32.3 

0.0 

30.1 

18.2 

27.1 

27.3 

27.4 

36.3 

198.7 

AMETHYST 

36.0 

30.1 

0.0 

34.5 

23.6 

25.0 

27.9 

20.4 

197.5 

GARNET 

40.5 

18.2 

34.5 

0.0 

21.3 

21.9 

22.0 

32.9 

191.3 

TOPAZ 

37.2 

27.1 

23.6 

21.3 

0.0 

18.6 

18.3 

22.2 

168.3 

EMERALD 

42.3 

27.3 

25.0 

21.9 

18.6 

0.0 

16.9 

21.2 

173.2 

DIAMOND 

46.4 

27.4 

27.9 

22.0 

18.3 

16.9 

0.0 

19.7 

178.6 

SAPPHIRE 

46.2 

36.3 

20.4 

32.9 

22.2 

21.2 

19.7 

0.0 

198.9 
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CHAPTER  4 

EVALUATOR  QUALIFICATIONS 


A  goal  of  the  NGCR  Program  in  general,  and  of  the  Operating  System 
Interface  Standard  selection  process  in  particular,  was  to  obtain  wide 
participation  by  the  industrial,  academic,  and  Navy  communities.  To  that  end, 
all  who  volunteered  to  participate  in  the  process  were  welcomed.  At  least  307 
different  individuals  responded  to  the  call  for  participation  by  attending  at 
least  one  meeting  of  the  OSSWG. 

Organizations  that  were  represented  at  a  minimum  of  two  meetings  by  the 
December  1989  meeting  were  invited  to  provide  two  official  evaluators  to 
participate  in  the  scoring  of  the  candidates.  More  than  200  were  qualified  as 
evaluators  on  this  basis,  and  letters  requesting  a  commitment  to  actively 
support  the  formal  evaluation  of  candidates  were  solicited  from  those  qualified. 
Altogether,  71  evaluators  submitted  letters  of  intent  to  participate.  These 
were  given  assignments  such  that  sufficient  evaluators  were  assigned  to  each 
aspect  of  the  evaluation. 

Table  4*1  shows  the  participation  in  the  evaluation  by  Navy  and  other 
(University/Industry /Non-Navy  Government)  organizations.  Appendix  C, 
Contributing  Organizations,  lists  all  organizations  represented  by  the  71 
•  evaluators  who  submitted  a  letter  of  commitment  and  received  an  evaluation 
assignment.  Not  all  were  able  to  complete  their  assignments.  The  names  of  the 
evaluators  who  actually  performed  the  scoring  of  the  candidates  are  provided  in 
Table  4-2.  Overall,  it  is  estimated  that  the  evaluation  itself  (from  screening 
process  through  selection)  encompassed  120  person-months  of  effort  over  a  period 
of  6  months. 


TABLE  4-1.  EVALUATION  PARTICIPANTS 

NAVY  OTHER 


INVOLVED  IN  ANY  WAY 

81 

226 

LETTERS  OF  COMMITMENT  RECEIVED 

25 

46 

EVALUATION  ASSIGNMENTS  COMPLETED 

21 

27 
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TABLE  4-2. 


PARTICIPANTS  IN  THE  CANDIDATE  SCORING  PROCESS 


CDR  Richard  Barbour 

SPAWAR  324 

Richard  Bargman 

NOSC 

Paul  Bicknell 

Mitre 

Richard  Brogan 

Booz,  Allen,  &  Hamilton 

Dala  Brouhard 

NOSC 

Gregory  Bussiare 

NUSC 

Antonio  Carangallo 

Mitre 

Gordon  Caswell 

ESL 

Thomas  Conrad 

NUSC 

B.  Dasarathy 

Concurrent  Computer 

Larry  Daubert 

Rockwell  International 

Iaobel  Davis 

Raytheon 

Steven  Davis 

DGM&S 

Dr.  Thomas  Drake 

Clemson  University 

Richard  Dvorchak 

Intel 

LT  Karl  Fairbanks 

NWC 

Gary  Fisher 

NIST 

Lester  Fraim 

Honeywell 

Dr.  Karen  Gordon 

IDA 

Dr.  Mars  Gralia 

JHU/AFL 

Daniel  Green 

NAVSWC 

Raymond  Gretlein 

Dynamics  Research 

Joseph  Gwinn 

Raytheon 

Barbara  Haleen 

Unisys 

James  Hall 

NIST 

Neil  Henderson 

Litton  Data  Systems 

Gail  Holmes 

NUSC 

Steven  Howell 

NAVSWC 

John  Johnson 

NAC 

Daniel  Juttelstad 

NUSC 

Kari  Kruempel 

Unisys 

Dr.  Jamas  Leathrum 

Clemson  University 

Michael  Linnig 

Texas  Instruments 

Dr.  Douglass  Locke 

IBM 

Warren  Loper 

NOSC 

Michael  Morgan 

Pacific  International  Center  for  High  Technology  Research  (PICHIR) 

Dr.  John  F.  Nixon 

General  Electric  Co.  Advanced  Technology  Laboratories 

Patricia  Oberodorf 

NADC 

James  Oblinger 

NUSC 

Frank  Prindle 

NADC 

John  Reed 

DEC 

Carl  Reinert 

Computer  Based  Systems 

Helmut  Roth 

NAVSWC 

Dr.  Timothy  Saponas 

Intel 

John  Shea 

NOS 

Del  Swanson 

Unisys 

Maria  Voreh 

NRL 

Patrick  Watson 

IBM 
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CHAPTER  5 
DATA  ACQUISITION 


This  chapter  documents  the  methods  used  to  collect  the  scoring  data  from 
the  various  industry  and  Navy  evaluators  of  the  seven  finalist  candidates. 


5.1  MOBILE  BRIEFING  RESULTS 

The  activities  and  successes  of  the  OSSWG  meeting  in  Mobile,  Alabama, 
22-26  January  1990,  are  described  in  this  section.  At  the  time  of  this  meeting, 
there  were  seven  final  candidates  identified. 

Before  the  December  OSSWG  meeting  there  had  been  ten  final  candidates. 
OSSWG  had  always  assumed  that  the  January  meeting  would  be  very  intensive, 
requiring  a  full  week  of  very  detailed  technical  presentations.  With  ten 
candidates  and  five  days  maximum  to  meet,  we  planned  that  each  candidate  would 
naturally  be  permitted  a  half  day  for  their  presentation.  With  seven  final 
candidates,  the  schedule  was  for  a  more  comfortable  three  and  a  half  days  of 
presentations  with  time  available  for  administrative  activities. 

The  format  required  of  each  candidate  presentation  was  specified  so  as  to 
assist  the  evaluators  in  filling  out  their  evaluation  forms  if  they  were  so 
inclined.  Specifically,  each  candidate's  advocate  was  to  format  his/her 
presentation  around  the  OSSWG  Requirements  Document.  For  all  service  classes 
he/she  was  to  provide  a  description  on  how  his/her  interface  supports,  or  fails 
to  support,  those  requirements.  Typically,  the  advocate  also  presented  how 
his/her  interface  supported  the  individual  criteria  that  comprised  the  service 
class.  It  was  also  required  that  each  advocate  show  the  linkage  between  their 
own  system  organization  and  that  documented  in  the  OSSWG  Reference  Model.  This 
helped  provide  a  frame  of  reference  between  terminology  used  by  OSSWG  and  that 
used  by  the  candidate. 
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After  taking  into  account  the  travel  requirements  and  time  constraints  of 
several  of  the  advocates,  the  following  agenda  was  set: 

Monday,  22  January  1990 

AM:  Registration rand  opening  discussions 

PM:  ORKID  candidate  presentation  by  Richard  Vanderlin 

from  Motorola. 

Tuesday,  23  January  1990 

AM:  POSIX  candidate  presentation  by: 

Jim  Isaak  from  DEC 

Fritz  Schulz  from  OSF 

Jim  Hall  from  NIST 

Steve  Carter  from  Bellcore 

Mike  Cossey  from  DOE  Oakridge 

Doug  Locke  from  IBM  Corp. 

Steve  Deller  from  Verdix  Corp. 

PM:  iRMX  candidate  presentation  by  Tim  Saponas  from  Intel 

Corp. 

Wednesday,  24  January  1990 

AM:  Open  but  candidate  material  was  made  available  for  the 

evaluators  to  begin  the  review  and  scoring  of  the 
candidates . 

PM:  Alpha  candidate  presentation  by  Doug  Jensen  from 

Concurrent  Computer  Corp. 

Thursday,  25  January  1990 

AM:  ARTX  candidate  presentation  by  Dave  Nelson-Gal  from 

Ready  Systems  Corp. 

PM:  Mach/TMach/RTMach  candidate  presentation  by: 

Brian  Boesch  from  DARPA 

Richard  Rashid  from  Carnegie  Mellon  University 

Hide  Tokuda  from  Carnegie  Mellon  University 

Steve  Walker  from  Trusted  Information  Systems 

Friday,  26  January  1990 

AM:  Cronus/SDOS  candidate  presentation  by  Jim  Berets 

from  BBN  Systems  and  Technologies  Corp. 
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These  presentations  proved  to  be  very  beneficial  to  the  evaluators  and 
substantially  aided  them  in  their  task  of  scoring  the  candidates.  Each  advocate 
focused  on  how  their  particular  system's  interface  addressed,  or  in  many  cases 
did  not  address,  each  of  the  OSSWG  requirements. 


5.2  EVALUATION  FORM  COLLECTION 

The  OSSWG  goal  was  to  assemble  enough  evaluators  so  that,  for  each  service 
class,  a  minimum  of  seven  evaluators  could  be  assigned  to  evaluate  the  seven 
candidate  interface  specifications.  The  OSSWG  determined  that  a  minimum  of 
seven  evaluators  per  service  class  was  required  to  provide  a  statistically 
sufficient  evaluation  of  each  candidate  specification.  The  requirement  that  an 
evaluator  evaluate  all  seven  candidate  specifications  for  a  service  class  was 
imposed  to  provide  a  fair  and  uniform  evaluation  process. 

OSSWG  members  who  qualified  were  asked  to  submit  a  letter  of  commitment  to 
perform  as  technical  evaluators  for  the  OSSWG  candidate  specification  evaluation 
process.  The  OSSWG  members  who  volunteered  as  technical  evaluators  were  also 
asked  to  list  the  service  classes  under  which  they  would  like  to  evaluate 
candidate  specifications,  based  on  personal  preference  and  areas  of  expertise. 
From  this  compiled  list,  the  Approach  Subgroup  made  the  final  evaluator 
assignments  with  the  following  ground  rules  in  mind:  whenever  possible,  assign 
an  evaluator  to  the  service  classes  of  his/her  preference;  have  at  least  seven 
evaluators  assigned  to  each  service  class;  and  do  not  overburden  any  evaluator 
with  assignments. 

The  Approach  Subgroup  was  able  to  assign  the  minimum  of  seven  evaluators 
to  each  service  class,  and  under  most  service  classes  more  than  seven  evaluators 
were  assigned.  In  addition  to  their  assigned  service  classes,  evaluators  were 
encouraged  to  evaluate  the  candidate  specifications  under  additional  service 
classes . 

A  complete  set  of  evaluation  forms,  instructions  for  completing  and 
submitting  the  forms,  the  OSSWG  Requirements  Document,  and  service  class 
assignments  as  well  as  the  documentation  on  the  seven  candidate  specifications 
were  distributed  to  the  technical  evaluators  via  the  United  States  Postal 
Service.  An  electronic  copy  of  the  evaluation  forms  was  also  made  available  to 
the  technical  evaluators  via  the  OSSWG  e-mail  distribution  list. 

Evaluators  were  allowed  to  submit  completed  evaluation  forms  by  one  of 
three  different  methods:  hand-written  or  typed  on  paper  copy  mailed  to  Naval 
Air  Development  Center  (NADC) ;  electronic  copy  e-mailed  to  a  special  DDN 
address;  or  interactively  via  an  interactive  evaluation  tool  provided  on  the 
NADC  computer.  The  original  due  date  for  submission  of  evaluation  forms  was  2 
February  1990  for  hand-written  and  9  February  1990  for  electronically  or 
interactively  submitted  forms.  Due  to  logistics  problems  with  the  distribution 
of  candidate  specification  documentation,  the  due  date  for  all  evaluation 
submissions  was  moved  back  to  28  February  1990. 
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Figure  5-1  provides  a  sample  section  of  a  blank  evaluation  form.  The 
evaluation  forms  contain  a  section  for  each  service  class  identified  in  the 
OSSWG  Requirements  Document.  These  evaluation  form  sections  are  further  broken 
into  subsections  which  correspond  (one-to-one)  to  the  interface  requirements  for 
the  subject  service  class.  For  each  subsection,  or  criterion,  the  evaluator  was 
required  to  supply  a  score  between  0  and  10  inclusive,  a  level  of  confidence  in 
the  score  given  (high,  medium,  or  low  confidence),  and  optionally  supporting 
rationale  for  the  score  given  and  any  additional  comments  the  evaluator  deemed 
necessary.  Evaluators  were  encouraged  to  reference  the  OSSWG  Requirements 
Document  for  descriptions  of  service  class  requirements  and  scoring  guidelines. 
In  general,  a  score  of  10  indicated  that  the  candidate  specification  fully 
supported  the  NGCR  OSIF  requirement  for  both  distributed  and  non- distributed 
environments,  while  a  score  of  0  indicated  that  the  requirement  was  not 
addressed/supported  by  the  specification. 
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*  OSSWG  OS  Interfaces  Evaluation 

* 

*  Project  Support  Environment  Interaction 

♦Service  Class: 

♦Evaluator  Name: 

♦Evaluator  ID: 

♦Candidate  ID: 


*10 . 1  Debug  Support 
♦Score  (0  -  10) : 

♦Confidence  Level  (H/M/L) : 

♦Rationale/References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line): 


♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line): 


*10.2  Execution  History 
♦Score  (0  -  10) : 

♦Confidence  Level  (H/M/L) : 

♦Rationale/References 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line): 
♦Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line): 


♦General  Comments 

♦(text;  don't  start  line  with  '*'  or  exceed  100  characters  per  line): 


♦End  of  evaluation  form  (Do  not  delete  this  line!)* 


FIGURE  5-1.  SAMPLE  OSSWG  SERVICE  CLASS  CRITERIA  EVALUATION  FORM 
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5.3  PROCESSING  OF  OSSWG  INTERFACE  EVALUATIONS 

The  major  tasks  in  the  data  collection  and  analysis  of  the  OSSWG  OS 
Interfaces  evaluation  were: 

o  Collecting  all  the  evaluation  forms  and  verifying  that  they  were 
complete . 

o  Parsing  the  evaluation  forms  to  produce  condensed  data  files, 
o  Processing  the  data  in  the  many  data  files  to  produce  tables  of 
results  and  graphics  data  files  that  could  be  downloaded  to  the 
spreadsheet  program. 

o  Producing  reports  on  incomplete  returns  of  the  evaluation  forms,  by 
evaluator,  by  service  class,  and/or  by  candidate. 

The  various  tools  were  developed  in  order  to  automate  the  processing  by 
speeding  the  process  and  insuring  correctness  of  results.  Some  of  the  common 
features  of  these  tools  include  the  following: 

o  All  the  data  files  in  the  ad  hoc  evaluation  data  base  have  the  date  and 
time  created  as  part  of  the  file  contents, 
o  Every  tool  records  its  actions  in  a  log  file.  The  information  logged 
briefly  identifies  the  tool,  including  a  time-stamp,  and  lists  the. 
actions  taken  and  any  errors  found. 

5.3.1  Interactive  Input  Tool 

5. 3. 1.1  Function.  This  tool  provides  a  simple  to  use  interactive  interface  for 
an  evaluator  to  record  her/his  scores,  comments,  etc.  The  tool  can  be  executed 
concurrently  by  multiple  users  (i.e.,  any  access  to  a  common  file/data  structure 
is  protected  from  corruption) .  The  tool  allows  the  user  to  review  and/or  edit 
an  evaluation  form  at  a  later  time.  Evaluator  passwords  are  required  and  the 
evaluation  data  is  kept  in  an  encrypted  form  to  make  it  difficult  for  a 
malicious  user  to  gain  information  about  other  evaluators'  scores  and  to  modify 
other  evaluators'  scores  without  being  detected. 

5. 3. 1.2  Formulas  Used,  (none) 

5. 3. 1.3  Input .  The  input  is  interactive  from  the  terminal  of  an  evaluator. 

The  terminal  is  not  required  to  have  cursor  positioning  or  other  "full  screen" 
capabilities. 

5. 3. 1.4  Output .  The  output  of  this  tool  is  an  evaluation  form  which  is  e- 
mailed  to  the  evaluator  and  to  the  account  designated  for  collecting  the 
evaluation  forms.  In  the  case  that  the  evaluator  does  not  complete  all  the 
criteria  for  a  service  class  before  exiting  the  tool,  the  scores  are  e-mailed  to 
the  evaluator  along  with  a  note  that  the  form  is  incomplete  and  has  not  been 
submitted;  no  incomplete  forms  should  be  e-mailed  to  the  collection  account. 
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5.3.2  Evaluation  Form  Parser 

5. 3. 2.1  Function,  This  tool  parses  evaluation  forms  produced  by  the 
interactive  evaluation  tool  as  well  as  those  e-mailed  in.  Multiple  service 
class  forms  are  split  into  individual  data  files;  e-mail  headers  are  stripped 
off;  the  forms  are  checked  that  all  criccTia  have  valid  scores  (  i.e.,  within 
valid  range)  and  that  the  user  ID  and  candidate  ID  are  valid.  The  tool  can 
process  more  than  a  single  file  within  one  execution  of  the  tool. 

5. 3. 2. 2  Formulas  Used,  (none) 

5. 3. 2. 3  Input .  The  input  consists  of  the  forms  e-mailed  in  and  produced  by  the 
interactive  evaluation  tool.  These  files  are  ASCII  files  which  may  contain 
e-mail  headers. 

5. 3. 2. 4  Output .  The  first  output  is  the  "Data  Only"  from  the  evaluation  forms 
which  is  entered  into  the  evaluation  data  base  if  and  only  if  the  form  is  valid. 
If  the  form  is  invalid  (or  incomplete),  then  the  errors  found  are  written  to  an 
error  report  file.  The  "Data  base"  is  a  collection  of  many  files  with  names 
which  identify  the  evaluator,  candidate  and  service  class  of  the  data  in  the 
file. 


The  second  output  consists  of  entries  in  the  log  file  which  identify  the 
specific  form  and  record  the  success/failure  of  the  parsing  operation. 

5.3.3  Evaluator  Reduction  and  Service  Class  Combiner 

5. 3. 3.1  Function.  In  general  this  tool  collects  all  the  data  from  the 
evaluation  data  base,  produces  reports  for  all  candidates,  and  generates  data 
files  for  producing  charts  with  a  graphics  spreadsheet.  For  each  criterion  the 
mean  of  the  scores  from  each  evaluator  as  well  as  the  maximum  score,  minimum 
score,  standard  deviation  and  the  estimated  confidence  are  calculated.  Also 
checks  are  made  that  there  are  enough  evaluators  for  each  service  class.  If  an 
evaluator  has  not  evaluated  the  minimum  number  of  candidates  (7)  for  a  specific 
service  class,  then  none  of  that  evaluator's  scores  are  included  in  the  results 
for  that  service  class.  (Note:  if  an  evaluator  has  not  scored  all  of  the 
criteria  for  a  service  class  that  form  would  be  rejected  by  the  form  parser  and 
none  of  the  scores  for  that  evaluation  form  would  be  in  the  data  base . ) 

5. 3. 3. 2  Formulas  Used.  See  Eq.  1,2, 3, 4, 5, 6, 7, 8, 9  in  Appendix  D,  Formulas  for 
Reduction  and  Analysis  of  Evaluation. 

5. 3. 3. 3  Input .  The  input  is  from  the  data  base:  the  scores  and  confidence 
values  for  each  criterion  for  each  evaluator  for  each  service  class  for  the 
specified  candidate  OS  interface. 
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5. 3. 3. 4  Output .  There  are  three  outputs  from  this  tool.  The  first  output  is 
the  results  report  (Appendix  E,  Criteria  Scores)  which  lists  for  each  criterion 
the  processed  score,  minimum  score,  maximum  score,  standard  deviation  and 
confidence  value.  The  results  report  also  includes  the  results  of  the  weighted 
average  of  scores  in  a  service  class  (using  weight  set  1)  and  the  results  of  the 
RAD  calculation  (using  weight  set  2) . 

The  second  output  is  a  set  of  chart  data  filos  from  which  the  spreadsheet 
program,  running  on  a  personal  computer,  can  produce  charts  of  the  results. 

The  third  output  is  an  entry  in  the  Evaluation  System  Log  File  which 
records  the  date  and  time,  the  candidate  processed  and  any  errors  found. 


5.3.4  Rationale- -Comments  Report  Generator 

5. 3. 4.1  Function.  These  tools  extract  the  rationales  or  comments  from  each  of 
the  evaluation  forms  along  with  the  ID  of  the  evaluator,  the  ID  of  the 
candidate,  the  service  class  and  the  criterion  number.  Separate  comments  and 
rationale  reports  are  produced  with  all  the  rationales/comments  for  one 
candidate  collected  together  by  criterion  and  service  class. 

5. 3. 4. 2  Formulas  Used.  (none) 

5. 3. 4. 3  Input .  The  input  is  the  evaluation  database. 

5. 3. 4. 4  Output.  The  comment  report  tool  produces  a  report  for  the  specified 

candidates  by  service  class  and  then  by  criteria  within  the  service  class  from 

the  comments  on  the  various  evaluation  forms .  All  the  comments  about  a 
particular  criterion  are  collected  together  along  with  information  which 
identifies  the  evaluator  who  submitted  each  comment.  Service  class  general 
comments  are  similarly  collected. 

The  second  tool  produces  the  report  which  is  the  collection  of  the 
rationales  for  each  individual  criterion.  This  report  is  very  similar  to  the 
first  one  except  that  there  is  no  general  rationale  for  a  service  class  and  this 
report  must  include  the  actual  score  as  well  as  the  evaluator  ID. 

5.3.5  Evaluator  Report  Generator  Tool 

5. 3. 5.1  Function.  This  tool  checks  the  database  to  report  those  evaluators  who 

have  not  submitted  a  complete  set  of  evaluation  forms.  The  general  rule  is 

that,  if  an  evaluator  submits  a  form  for  one  candidate  for  one  service  class, 
he/she  is  required  to  submit  evaluation  forms  for  all  OS  Interface  candidates 
for  that  service  class  in  order  for  any  of  his/her  scores  to  be  counted. 

5. 3. 5. 2  Formulas  Used.  (none) 
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5. 3. 5. 3  Input .  The  input  for  this  tool  is  the  evaluation  database.  Actually  a 
list  of  the  file  names  in  the  database  directory  is  used  to  determine  which 
evaluators  have  not  completed  which  service  classes. 

5. 3. 5. 4  Output .  The  output  report  from  this  tool  lists  the  evaluators  (or 
their  IDs)  and  the  service  classes  for  which  they  have  submitted  complete  and 
incomplete  sets  of  forms. 

5.3.6  Candidate  Report  Generator  Tool 

5. 3. 6.1  Function.  This  tool  checks  the  database  to  report  those  candidate  OS 
Interfaces  which  do  not  have  a  complete  set  of  evaluation  forms  submitted.  The 
report  lists  the  number  of  evaluators  for  each  service  class  for  each  candidate. 

5. 3. 6. 2  Formulas  Used.  (none) 

5. 3. 6. 3  Input .  The  input  for  this  tool  is  the  evaluation  database.  A  list  of 
the  files  in  the  database  directory  is  used  to  determine  which  evaluators  have 
not  completed  which  service  classes. 

5. 3. 6. 4  Output .  The  output  report  lists  the  candidates  and  for  each  the  number 
of  evaluators  for  each  service  class  that  have  completed  evaluation  forms. 

5.3.7  Generation  of  Figures 

The  results  of  the  evaluation  are  downloaded  into  a  graphics  spreadsheet 
to  produce  the  many  charts  of  the  results. 

Figures  5-2,  5-3,  and  5-4  show  the  dataflow  through  the  evaluation  processing. 
Each  of  the  tools  (programs)  described  in  sections  5.3.1  -  5.3.6  is  described  in 
more  detail  below. 
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Figure  5-2.  DATA  COLLECTION  -  UNIX 
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FIGURE  5-4.  EVALUATION  DATAFLOW  -  MACINTOSH 
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CHAPTER  6 
SUMMARY  RESULTS 


The  results  documented  below  were  derived  from  the  raw  scores  provided  by 
the  evaluators,  according  to  the  averaging,  weighting,  and  combining  processes 
described  in  the  Evaluation  Process  Report  and  Chapter  5  of  this  report.  The 
results  are  generally  categorized  as  either  "technical"  or  "programmatic"  in 
nature.  The  technical  results  derive  from  scoring  the  candidates  against 
criteria  defined  in  the  Requirements  Document  and  are  presented  in  section  6.1 
below.  The  programmatic  results  derive  from  scoring  the  candidates  on 
programmatic  issues  defined  in  the  Evaluation  Process  Report  and  are  presented 
in  section  6.2  below.  See  Appendix  E  (Criteria  Scores)  for  a  summary  of  the 
results . 

Several  formats  are  used  to  present  the  results,  including  both  tables  and 
charts.  Tabular  data  includes  average  score,  standard  deviation,  an  evaluator 
confidence  measure,  and  maximum  and  minimum  scores.  The  scope  of  the  raw  data 
on  which  these  results  are  based  is  as  follows.  A  single  complete  set  of  raw 
scores  for  one  candidate  consists  of  193  data  points  (8  Programmatic  Issue 
scores,  29  scores  on  general  technical  criteria,  and  156  scores  on  specific 
technical  criteria).  Table  6-1  shows  a  summary  of  the  data  collected.  At  least 
7  independent  sets  of  scores  were  obtained  for  each  criterion  for  each 
candidate.  For  many  subsets  of  the  evaluation  criteria  many  more  than  7  sets  of 
scores  were  obtained.  As  a  result,  the  total  number  of  data  points  for  all 
candidates  was  19,964. 


6-1 


NAVSWC  TR  90-246 


TABLE  6-1.  EVALUATION  DATA  BASE 

Data  Points  Per 

#  of  #  of  Service  Class  Per 

Class  #  Evaluators  Criteria  Candidate 


0 

20 

8 

160 

1 

45 

29 

1305 

2 

9 

1 

9 

3 

7 

24 

168 

4 

9 

1 

9 

5 

13 

6 

78 

6 

8 

20 

160 

7 

8 

11 

88 

8 

11 

10 

110 

9 

9 

14 

126 

10 

10 

2 

20 

11 

12 

14 

168 

12 

11 

12 

132 

13 

11 

11 

121 

14 

9 

3 

27 

15 

13 

7 

91 

16 

9 

20 

180 

Totals  193  2852 

2852  Data  Points  *  7  Candidates  -  19,964  Total  Data  Points 
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6.1  TECHNICAL  EVALUATION 

The  technical  evaluation  was  performed  by  both  Navy  and  non-Navy  evaluators 
based  on  the  requirements  defined  in  the  OSSWG  Requirements  Document,  Version 
2.0.  That  document  identifies  the  16  technical  classes  of  operating  system 
services.  Of  these,  15  relate  to  specific  services  that  an  operating  system 
interface  must  provide.  The  other  class  contains  a  collection  of  general 
requirements  in  an  operating  system  interface.  Each  service  class  reflects  a 
number  of  specific  evaluation  criteria  on  which  the  candidates  were  judged. 

Table  6-2  indicates  the  number  of  evaluation  criteria  associated  with  each 
class. 

TABLE  6-2.  NUMBER  OF  CRITERIA  PER  TECHNICAL  SERVICE  CLASSES 


SERVICE  CLASS  NUMBER  OF  CRITERIA 


1.  GENERAL  REQUIREMENTS 

29 

2.  ARCHITECTURE  DEPENDENT  INTERFACES 

1 

3.  CAPABILITY  AND  SECURITY  INTERFACES 

24 

4.  DATA  INTERCHANGE  INTERFACES 

1 

5.  EVENT  AND  ERROR  INTERFACES 

6 

6.  FILE  INTERFACES 

20 

7.  GENERALIZED  I/O  INTERFACES 

11 

8.  NETWORK  AND  COMMUNICATIONS  INTERFACES 

10 

9.  PROCESS  MANAGEMENT  INTERFACES 

14 

10.  PROJECT  SUPPORT  ENVIRONMENT  INTERFACES 

2 

11.  RELIABILITY,  ADAPTABILITY,  MAINTAIN¬ 
ABILITY  INTERFACES 

14 

12.  RESOURCE  MANAGEMENT  INTERFACES 

12 

13.  SYNCHRONIZATION  AND  SCHEDULING  INTERFACES 

11 

14.  SYSTEM  INITIALIZATION  AND  REINIT¬ 
IALIZATION  INTERFACES 

3 

15.  TIME  SERVICES  INTERFACES 

7 

16.  ADA  LANGUAGE  SUPPORT  INTERFACES 

20 
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A  difference  between  service  class  1  (General)  and  the  remaining  service 
classes  is  that  the  criteria  which  compose  service  class  1  are  independent  of 
each  other.  Therefore  it  is  not  reasonable  to  generate  an  overall  service  class 
score  for  service  class  1.  For  this  reason,  the  specific  and  the  general 
service  class  scores  are  presented  in  separate  sections  below.  Eight  RADs  were 
defined  in  the  Evaluation  Process  Report  along  with  weights  relating  the  overall 
scores  on  the  specific  service  classes  to  a  score  for  each  application  domain. 

Consequently,  Section  6.1.1  below  provides  the  results  for  the  specific 
service  classes.  Section  6.1.2  reports  the  effect  of  mapping  the  specific 
service  class  scores  onto  the  eight  representative  application  domains.  Section 
6.1.3  reports  the  results  for  the  general  service  class. 

6.1.1  Service  Classes 


The  following  tables  and  figures  summarize  the  outcome  of  the  evaluation 
by  service  class.  Table  6-3  presents  a  summary  of  results  including  the  score 
and  error  approximation  (sigma)  for  each  candidate  against  each  service  class. 
Figures  6-l(a),  6-l(b),  and  6-l(c)  depict  the  information  in  graphical  form. 


TABLE  6-3.  SUMMARY  OF  CLASS  SCORES 


Note:  scores  and  standard  deviations  (sigmas)  are  both  listed:  score | sigma 
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FIGURE  6-1 (a).  WEIGHTED  CLASS  SCORES  FOR  EACH  CANDIDATE 
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FIGURE  6-1 (b).  WEIGHTED  CLASS  SCORES  FOR  EACH  CANDIDATE 
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FIGURE  6-1 (C).  WEIGHTED  CLASS  SCORES  FOR  EACH  CANDIDATE 
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6.1.2  Representative  Application  Domain  Results 

The  following  tables  and  figures  summarize  the  outcome  of  the  evaluation 
by  RAD.  Table  6-4  presents  a  summary  of  results  including  the  score  and  error 
approximation  (sigma)  for  each  candidate  against  each  RAD.  A  sigma  is  one 
standard  deviation.  Figure  6-2  depicts  the  information  in  graphical  form. 


TABLE  6-4.  SUMMARY  OF  RAD  RESULTS 


RAD  ALPHA  ARTX  CRONUS  IRMX  MACH  ORKID  POSIX 


Amethyst  5.88 

0.47  3.59 

0.38  3.80 

0.38  4.91 

0.38  3.80 

0.43  1.90 

0.26  4.93 

0.38 

Diamond  6 . 14 

0.50  3.76 

0.38  3.36 

0.44  5.22 

0.38  3.58 

0.46  2.16 

0.33  4.77 

0.46 

Emerald  5.82 

0.49  3.48 

0.38  3.48 

0.42  4.92 

0.37  3.77 

0.45  1.91 

0.31  4.74 

0.44 

Garnet  6.09 

0.50  3.67 

0.38  3.15 

0.43  5.20 

0.39  3.37 

0.45  2.07 

0.33  4.71 

0.46 

Opal  6.21 

0.53  3.67 

0.41  3.50 

0.45  5.17 

0.41  3.37 

0.48  2.18 

0.34  4.69 

0.47 

Ruby  5.86 

0.50  3.98 

0.41  3.72 

0.40  5.19 

0.39  3.89 

0.46  1.72 

0.27  5.22 

0.39 

Sapphire  6.02 

0.46  3.74 

0.37  3.49 

0.38  5.10 

0.37  3.81 

0.42  2.08 

0.28  4.85 

0.38 

Topaz  5 . 94 

0.52  3.79 

0.42  3.53 

0.43  5.16 

0.41  3.62 

0.48  1.89 

0.31  4.73 

0.44 

Note:  scores  and  standard  deviations  (sigmas)  are  both  listed:  score | sigma 
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FIGURE  6-2.  REPRESENTATIVE  APPLICATION  DOMAIN  RESULTS 
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6.1.3  General  Technical  Issues 


The  evaluation  criteria  for  the  OSSWG  baseline  candidates  included  a 
collection  of  issues  which  were  sufficiently  pervasive  to  defy  inclusion  in  the 
already  defined  service  classes.  These  criteria  were  brought  together  into  a 
special  class  called  General  Requirements  (Service  Class  1) .  Not  only  were 
these  criteria  pervasive,  they  were  also  somewhat  pedestrian.  Thus,  they  could 
reasonably  be  evaluated  by  all  evaluators  regardless  of  the  evaluator's  special 
expertise.  Unlike  the  other  service  classes,  the  criteria  scores  in  this  class 
were  collected  from  a  large  number  of  evaluators  (i.e.,  45). 

Having  little  connection  with  each  other,  the  criteria  within  the  General 
Requirements  service  class  do  not  lend  themselves  to  averaging  within  the 
service  class.  For  this  reason,  weights  were  not  generated  for  the  criteria, 
and  the  results  from  the  29  criteria  within  this  service  class  are  only 
presented  separately  with  no  suggestion  of  connections  or  aggregations.  This 
information  is  in  tabular  form  in  Table  6-5  and  graphical  form  in  Figures 
6-3(a) ,  6-3(b) ,  6-3(c),  6-3(d),  6-3(e),  and  6-3(f). 
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TABLE  6-5.  CANDIDATE  RESULTS  FOR  EACH  GENERAL  REQUIREMENT 
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FIGURE  6-3 (a).  GENERAL  SERVICE  CLASS  BY  CRITERION 
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FIGURE  6-3 (b) .  GENERAL  SERVICE  CLASS  BY  CRITERION 
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FIGURE  6-3 (c).  GENERAL  SERVICE  CLASS  BY  CRITERION 
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FIGURE  6-3 (d) .  GENERAL  SERVICE  CLASS  BY  CRITERION 
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FIGURE  6-3 (f).  GENERAL  SERVICE  CLASS  BY  CRITERION 
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6.2  PROGRAMMATIC  EVALUATION  • 

The  Programmatic  Requirements  are  described  in  the  Evaluation  Process 
report.  There  are  eight  programmatic  requirements.  These  were  formulated  to 
address  topics  that  were  relevant  to  the  selection  process  but  were  not  a  part 
of  the  proper  technical  requirements  for  an  operating  system  interface.  They 
are  intended  to  capture  the  other  factors  which  are  important  to  the  Navy  in 
this  selection. 

The  programmatic  requirements  were  evaluated  by  all  of  the  Navy  personnel. 
The  results  of  these  evaluations  are  shown  in  the  table  and  chart  below. 
Throughout  most  of  the  evaluation  and  analysis,  the  programmatic  requirements 
were  handled  independently  of  the  technical  ones,  with  the  technical  ones  being 
considered  first. 

Table  6-6  gives  the  average  of  the  evaluators'  scores  for  each  candidate 
for  each  of  the  eight  criteria,  along  with  one  standard  deviation.  Figure  6.4 
graphs  these  scores,  including  the  variation  represented  by  the  standard 
deviations  (sigmas) . 


TABLE  6-6.  PROGRAMMATIC  CRITERIA  RESULTS 


Criteria  ALPHA  ARMX  CRONUS  IRMX  MACH  ORKID  POSIX 


1  7.50 

2.16  5.45 

1.28  7.40 

1.98  5.45 

1.19  8.40 

1.73  7.90 

2.88  9.45 

2.24 

2  6.10 

2.53  5.10 

2.20  5.90 

1.97  5.55 

2.44  4.40 

2.58  5.60 

2.72  6.05 

2.04 

3  5.45 

1.90  8.15 

1.84  7.00 

2.00  7.80 

1.77  6.75 

1.59  4.05 

2.35  5.95 

2.26 

4  6.75 

2.05  8.00 

1.75  7.05 

1.96  8.70 

1.56  6.05 

2.01  6.40 

2.28  7.95 

1.70 

5  4.75 

1.68  6.20 

2.86  4.80 

2.61  6.30 

2.62  6.05 

2.19  5.05 

1.96  8.75 

2.29 

6  6.70 

2.25  8.25 

1.83  7.75 

1.97  8.15 

1.93  6.80 

1.82  6.95 

2.11  6.40 

2.54 

7  5.45 

2.31  6.45 

1.47  5.45 

2.46  7.40 

2.04  5.55 

1.90  7.50 

2.44  8.85 

1.35 

8  5.40 

2.93  5.80 

2.59  6.00 

2.58  5.60 

2.44  5.60 

2.85  6.05 

2.48  6.80 

2.33 

Note:  scores  and  standard  deviations  (sigmas)  are  both  listed:  score  sigma 
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CHAPTER  7 

ALTERNATIVE  VIEWS  OF  DATA 


Upon  examination  of  the  data,  it  became  apparent  that  some  additional 
analyses  were  necessary.  This  conclusion  was  driven  by  three  circumstances: 

A.  The  RAD  scores  did  not  show  the  differentiations  between  the 
candidates  that  its  creators  had  anticipated;  this  situation  needed 
to  be  understood. 

B.  The  scores  of  individual  evaluators  for  a  given  criterion  often 
spanned  the  full  range  from  0  to  10,  as  reflected  in  the  values  of 
the  error  estimates;  this  needed  investigation  to  assure  the 
validity  of  the  data. 

C.  The  initial  results  from  the  data  showed  three  candidates  scoring 
particularly  high,  but  still  relatively  close  to  one  another;  this, 
when  coupled  with  the  circumstance  (B)  above,  led  to  a  desire  to 
examine  and  compare  these  three  candidates  at  a  greater  level  of 
detail  in  order  to  more  effectively  determine  the  true  "leader." 


7.1  GLASS  DATA 

The  unexpected  similarity  of  the  "RAD  Scores"  for  each  candidate  resulted 
in  a  flurry  of  reanalysis  and  sensitivity  analysis  using  synthetic  extreme-case 
RADs.  None  of  these  efforts  had  any  practical  effect  on  the  results.  Even  the 
extreme  RADs  failed  to  clearly  differentiate  among  the  candidates. 

Consequently,  a  new  RAD,  "GLASS,"  with  uniform  weights,  was  created.  As 
expected,  the  GLASS  RAD  scores  were  in  the  middle  of  the  pack  as  can  be  seen  in 
Table  7-1  and  Figure  7-1. 
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TABLE  7-1.  SUMMARY  OF  RAD  RESULTS  WITH  GLASS  RAD 


RAD 

ALPHA 

ARTX 

CRONUS 

IRMX 

MACH 

ORKID 

POSIX 

Amethyst 

5.88 

3.59 

3.80 

4.91 

3.80 

1.90 

4.93 

Diamond 

6.14 

3.76 

3.36 

5.22 

3.58 

2.16 

4.77 

Emerald 

5.82 

3.48 

3.48 

4.92 

3.77 

1.91 

4.74 

Garnet 

6.09 

3.67 

3.15 

5.20 

3.37 

2.07 

4.71 

Opal 

6.21 

3.67 

3.50 

5.17 

3.37 

2.18 

4.69 

Ruby 

5.86 

3.98 

3.72 

5.19 

3.89 

1.72 

5.22 

Sapphire 

6.02 

3.74 

3.49 

5.10 

3.81 

2.08 

4.85 

Topaz 

5.94 

3.79 

3.53 

5.16 

3.62 

1.89 

4.73 

Glass 

5.98 

3.88 

3.53 

5.21 

3.82 

1.91 

4.93 
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FIGURE 
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Note  that  the  RAD  weights  (weight  set  2)  are  applied  to  the  raw  candidate 
evaluation  scores  as  aggregated  by  service  class.  Thus,  the  manipulations 
carried  out  here  do  not  change  any  of  the  actual  scores  but  merely  serve  as 
alternate  interpretations  of  those  scores .  The  GLASS  RAD  represents  an 
application  domain  in  which  all  service  classes  are  of  equal  importance. 

7.2  RAD  ANALYSIS  OF  THE  GENERAL  CRITERIA  SCORES 

In  an  effort  to  investigate  the  conjecture  that  some  of  the  "General 
Requirements"  (i.e.,  Service  Class  1)  were  potentially  sensitive  to  the 
application  domain,  these  scores  were  projected  onto  the  RADs.  Of  the  29 
original  general  criteria,  twelve  were  selected  as  the  technical  subset  for  the 
purpose  of  further  analysis.  The  selected  criteria  were: 

2.1.4  Architecture  Independence 

2.1.10  Ada  Language  Binding  Syntax 

2.1.20  Reaction  to  Blocking  Services 

2.1.21  Bounding  Operating  Systems  Service  Times  and 
Context  Switching 

2.1.22  Configurability 

2.1.23  Transaction  Scheduling  Information 

2.1.24  Access  Control 

2.1.25  Transparency 

2.1.26  Resilience 

2.1.27  Network  Partition 

2.1.28  Reference 

2.1.29  Reallocation 

as  with  all  the  other  criteria,  the  individual  evaluator  scores  were 
averaged  over  the  evaluators  for  each  of  these  "Technical  General  Criteria."  A 
set  of  weights  expressing  the  importance  of  each  of  these  criteria  to  the  RADs 
was  generated  by  the  Approach  Subgroup  after  the  raw  data  gathering  was  nearly 
complete.  These  weights  were  purposely  biased  toward  extreme  values  in  the 
hopes  of  overcoming  some  of  the  difficulties  already  encountered  with  the  RAD 
scores.  The  results  are  shown  in  Tables  7-2,  7-3  and  7-4  and  Figures  7-2  and  7- 
3.  The  scores  listed  are  calculated  in  the  same  way  as  the  RAD  scores.  Table  7- 
3  and  Figure  7-2  show  the  results  using  only  the  General  Requirements  Service 
Class  weights  of  Table  7-2.  Table  7-4  and  Figure  7-3  show  the  results  of  using 
both  the  General  Requirements  Service  Class  weights  along  with  Weight  Set  2. 
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TABLE  7-2.  STANDARD  SERVICE  CLASS  WEIGHTS  FOR  RAD  MAPPING 


R 

O 

A 

G 

T 

E 

u 

S 

1.4 

10 

5 

10 

10 

10 

10 

10 

10 

1.10 

0 

5 

10 

10 

10 

10 

10 

10 

1.20 

5 

10 

10 

10 

10 

10 

5 

10 

1.21 

0 

10 

10 

10 

10 

10 

10 

10 

1.22 

0 

10 

5 

10 

0 

0 

10 

5 

1.23 

0 

0 

10 

5 

10 

5 

0 

10 

1.24 

10 

5 

10 

0 

5 

10 

0 

10 

1.25 

10 

0 

10 

0 

10 

0 

0 

10 

1.26 

5 

0 

10 

0 

10 

10 

10 

10 

1.27 

5 

0 

10 

0 

10 

10 

10 

10 

1.28 

5 

0 

10 

0 

10 

10 

10 

10 

1.29 

5 

0 

10 

0 

10 

10 

10 

10 

TABLE  7-3.  RAD  SCORES  BASED  ONLY  ON  GENERAL  SERVICE  CLASS  CRITERIA  (G-RADS) 


RAD  ALPHA  ARTX  CRONUS  IRMX  MACH  ORKID  POSIX 


Amethyst  7.64 

0.71  4.85 

0.75  5.31 

0.75  6.28 

0.71  4.56 

0.92  3.81 

0.76  5.38 

0.79 

Diamond  7.28 

0.94  5.76 

0.91  5.35 

0.92  6.10 

0.91  4.59 

1.09  4.00 

1.00  5.79 

0.97 

Emerald  7.42 

0.79  4.71 

0.83  4.97 

0.84  6.13 

0.79  4.48 

0.98  3.48 

0.83  5.45 

0.86 

Garnet  7.28 

1.08  6.85 

1.03  4.89 

1.10  6.35 

1.01  4.67 

1.34  4.24 

1.10  6.22 

1.12 

Opal  7.56 

1.13  6.37 

1.00  5.23 

1.18  6.73 

1.02  4.71 

1.40  4.38 

1.24  6.16 

1.17 

Ruby  8.21 

0.74  4.22 

0.91  6.91 

0.88  6.60 

0.82  5.33 

1.16  4.77 

0.99  5.87 

0.87 

Sapphire  7.64 

0.71  4.85 

0.75  5.31 

0.75  6.28 

0.71  4.56 

0.92  3.81 

0.76  5.38 

0.79 

Topaz  7.61 

0.73  4.90 

0.80  5.13 

0.78  6.17 

0.75  4.50 

0.96  3.80 

0.80  5.32 

0.83 

Glass  7.64 

0.71  4.96 

0.74  5.36 

0.75  6.34 

0.70  4.56 

0.92  3.86 

0.77  5.38 

0.78 

Note:  scores  and  standard  deviations  (sigmas)  are  both  listed: 


score | sigma 
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TABLE  7-4.  EXTENDED  REPRESENTATIVE  APPLICATION  DOMAINS  (E-RADS) 

(INCLUDES  SELECTED  SERVICE  CLASS  1  CRITERIA) 


RAD  ALPHA  ARTX  CRONUS  IRMX  MACH  ORKID  POSIX 


Amethyst  6.84 

0.44  4.28 

0.45  4.62 

0.44  5.66 

0.42  4.22 

0.54  2.94 

0.43  d.x8 

0,46 

Diamond  6 . 64 

0.50  4.63 

0.45  4.23 

0.47  5.61 

0.45  4.02 

0.54  2.97 

0.48  5.21 

0.50 

Emerald  6.63 

0.47  4.10 

0.46  4.23 

0.47  5.53 

0.44  4.13 

0.54  2.70 

0.45  5.10 

0.49 

Garnet  6.59 

0.54  5.01 

0.49  3.88 

0.53  5.68 

0.48  3.92 

0.62  2.99 

0.50  5.35 

0.54 

Opal  6 . 74 

0.55  4.73 

0.47  4.18 

0.54  5.78 

0.47  3.90 

0.62  3.04 

0.53  5.27 

0.54 

Ruby  6.88 

0.43  4.09 

0.46  5.11 

0.45  5.80 

0.42  4.52 

0.57  3.05 

0.46  5.50 

0.44 

Sapphire  6.86 

0.43  4.32 

0.43  4.44 

0.43  5.72 

0.41  4.20 

0.52  2.98 

0.42  5.13 

0.45 

Topaz  6.83 

0.46  4.38 

0.47  4.38 

0.46  5.70 

0.45  4.09 

0.56  2.91 

0.45  5.04 

0.49 

Glass  6.71 

0.42  4.36 

0.39  4.35 

0.41  5.71 

0.38  4.15 

0.48  2.78 

0.38  5.13 

0.42 

Note:  scores  and  standard  deviations  (sigmas)  are  both  listed:  score | sigma 
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FIGURE  7-2.  REPRESENTATIVE  APPLICATION  DOMAIN  -  GENERAL  SERVICE  CLASS  ONLY 
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REPRESENTATIVE  APPLICATION  DOMAIN  RESULTS  -  ALL  TECHNICAL  SERVICE  CLASS 
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7.3  PROGRAMMATIC  ISSUES  RANKINGS 

The  eight  programmatic  requirements  were  initially  equally  weighted  (i.e., 
unweighted).  However,  it  was  clear  to  the  OSSWG  membership  that  some  were  far 
more  important  to  the  success  of  the  effort  than  others.  In  order  to  devise 
weights  for  them,  the  requirements  were  initially  ranked  in  order  of  importance. 
That  ranking  and  its  rationale  follow.  (The  references  in  parentheses  below  are 
to  the  sections  in  the  Evaluation  Process  report  in  which  the  original 
description  of  the  programmatic  requirements  can  be  found.) 

#1  Public  Domain  Interfaces  (3.2.1) 

"In  the  Public  Domain"  is  a  founding  precept  for  the  NGCR  view  of  the 
world.  It  allows  free  access  to  the  interface  and  must  remain  as  a  top  priority 
for  the  Navy  and  this  program.  Additionally,  lack  of  public  domain  interfaces  is 
grounds  for  de-selection  of  an  interface.  A  public  domain  interface  is 
important  because  it  encourages  the  ultimate  goal  of  "openness"  in  the  sense 
that  any  vendor  can  plug  in  or  build  on  to  it.  The  OSSWG  did  not  specify  this 
ultimate  goal  of  openness  in  its  evaluation  criteria.  This  criteria  is  a  first 
step  to  openness. 

#2  Navy  Influence  (3.2.2) 

The  Navy  wants  to  influence  the  interface  definition  selected.  Any 
selected  baseline  upon  which  to  pursue  a  standardization  effort  must  be  able  to 
accommodate  present  and  future  Navy  needs. 

The  Navy  is  "riding  industry  trends"  so  it  must  be  careful  how  much  it 
attempts  to  dictate  the  standardization  process.  Too  much  influence  and  the 
Navy  will  not  achieve  the  hoped-for  commercial  acceptance.  If,  on  the  other 
hand,  the  candidate  does  not  answer  the  Navy  user's  needs  and/or  the  Navy  cannot 
effectively  influence  the  candidate  to  accomplish  this,  then  what  has  been 
accomplished? 

#3  Commercial  Acceptance  (3.2.5) 

The  emphasis  is  on  wide  acceptance.  Through  acceptance  of  broad-based 
industry  supported  standards  and  products,  the  Navy  will  ensure  an  industrial 
base  capable  of  supporting  its  computing  needs  in  an  era  of  continuing 
accelerated  technological  progress  during  declining  DoD  funding.  This  has 
evolved  into  being  a  primary  consideration  of  not  just  the  OSSWG  but  of  the  NGCR 
Program. 

#4  Economics/Cost  (3.2.8) 

Economics  and  acceptance  are  intricately  tied  together.  For  example,  an 
important  consideration  is  industry's  perception  as  to  whether  it  can  develop  a 
cost  effective  implementation  of  the  standard. 


7-9 


NAVSWC  TR  90-246 


#5  Maturity /Confidence  (3.2.3) 

This  criterion  indicates  whether  the  candidate  has  implementations, 
whether  implementations  are  planned,  etc.,  and  the  degree  of  Navy  confidence 
that  the  wide  variety  of  Navy  implementations  could  be  achieved. 

#6  Timeframe  (3.2.6) 

The  original  rationale  given  in  3.2.6  applies  here  as  well:  Can  a 
standard  be  achieved  with  the  candidate  in  the  timeframe  of  the  OSSWG?  This  one 
is  somewhat  "elastic"  from  the  perspective  of  the  Program  Management  Office 
(PMO)  itself,  i.e.,  it  is  somewhat  dependent  on  the  resources  the  PMO  is  willing 
to  dedicate  (to  compress,  influence,  etc.),  or  the  PMO  can  choose  to  allow  the 
schedule  to  "stretch  out." 

#7  User  Influence  (Historical)  (3.2.7) 

The  original  intent  here  was  to  assess  the  amenability  of  the  candidate  to 
influence.  It  carries  little  weight  for  actually  selecting  the  candidate 
baseline  now.  The  addition  of  the  parenthetical  note,  "Historical,"  in  the 
title  is  intended  to  reiterate  that  this  "user  influence"  is  talking  about  how 
the  candidate  has  reacted  to  its  users  in  the  past;  it  is  not  trying  to  capture 
how  amenable  it  will  be  to  user  (i.e.,  Navy)  influence  in  the  future,  which  is 
the  focus  of  the  "Navy  Influence"  requirement. 

#8  Documentation  (3.2.4) 

We  have  what  we  have.  It  helped  provide  the  evaluation  data.  Its 
potential  as  a  discriminator  for  baseline  selection  is  minimal. 

After  reaching  agreement  on  this  relative  ranking  of  the  programmatic 
requirements,  consideration  was  given  to  the  weight  set  which  should  be  applied 
in  order  to  derive  scores  for  Service  Class  0.  Several  alternatives  were 
examined,  but  all  had  approximately  the  same  effect.  A  weight  set  in  which  the 
first  4  have  weights  of  10  each,  the  next  2  have  weights  of  8  each  and  the  last 
2  have  weights  of  5  each.  The  results  using  these  weights  are  shown  in  Table  7- 
5  below. 
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TABLE  7-5.  PROGRAMMATIC  RESULTS  WEIGHTED  AVERAGES 

POSIX  7.50 
IRMX  6.62 
ARTX  6 . 50 
CRONUS  6.39 
MACH  6 . 23 
ORKID  6.11 
ALPHA  6 . 00 


7.4  EFFECT  OF  NON- WEIGHTED  (AVERAGE-WEIGHTED)  SCORES 

Because  of  the  newness  of  this  type  of  evaluation  and  the  various  types 
of  scoring  (weighting)  methods  and  algorithms  being  tried,  it  was  requested 
that  an  unweighted  average  method  be  applied  to  the  criteria  scores  in  order  to 
ascertain  any  differences  between  this  and  other  methods.  The  criteria  scores 
were  summed  and  normalized.  Three  averages  were  computed  for  each  candidate: 

(1)  average  score  on  all  193  criteria 

(2)  average  score  on  all  185  technical  criteria 

(3)  average  score  on  the  8  programmatic  issues 

The  results  are  depicted  in  Table  7-6.  The  net  result  of  this  effort  was 
that,  weighted  or  averaged,  the  leading  candidates  remained  the  same. 

TABLE  7-6.  UNWEIGHTED  RAW  SCORE  AVERAGES 


ALPHA 

ARTX 

CRONUS 

IRMX 

MACH 

ORKID 

POSIX 

ALL  CRITERIA 
Average 

Sigma 

6.01 

2.02 

4.51 

3.20 

3.88 

2.72 

5.62 

3.00 

4.58 

2.50 

2.56 

2.86 

5.96 

2.65 

TECHNICAL  CRITERIA 
Average  6.01 

Sigma  2.06 

4.41 

3.23 

3.77 

2.72 

5.57 

3.04 

4.51 

2.52 

2.40 

2.80 

5.89 

2.67 

PROGRAMMATIC 

Average 

Sigma 

CRITERIA 

6.01 

0.91 

6.68 

1.28 

6.79 

1.19 

6.87 

1.30 

6.20 

1.17 

6.19 

1.28 

7.53 

1.39 
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7.5  RANK  ORDERING  DATA 

Using  several  views  of  the  statistical  results,  unweighted  rank  ordering 
of  the  candidates  was  applied  in  an  effort  to  reveal  a  foundation  for 
discrimination.  The  three  views  deemed  pertinent  were: 

o  Ranking  by  the  "tops  in  service  class" --indicating  both  the  quantity 
and  identification  of  the  specific  service  classes  in  which  each 
candidate  was  considered  top  ranked,  without  regard  for  the 
programmatic  issues. 

o  Ranking  based  on  "win,  place,  and  show" --charting  the  number  of 
occurrences  of  each  candidate's  relative  positioning  by  service 
class,  e.g.,  4  first  place,  2  seconds,  6  thirds.  Programmatic 
scoring  not  included. 

o  Ranking  order  range  of  effectiveness- -defining  the  range,  by  service 
class,  for  which  scores  remain  in  an  effective  range  (i.e.,  7  or 
larger) .  This  would  enable  constraining  the  ordering  to  only  those 
candidates  having  effectively  met  the  requirements  of  a  specific 
service  class  and  the  programmatic  issues. 


7.5.1  Top  Ranked  Data 

The  candidates  were  ranked  by  their  superior  evaluation  in  the  service 
classes.  These  service  classes  are  listed  in  Table  6-2.  Table  7-7  gives  a 
ranking  of  the  candidates  according  to  the  greatest  number  of  service  classes  in 
which  that  candidate  scored  highest.  Service  class  0  (programmatics)  is  not 
considered  in  this  ranking. 


TABLE  7-7.  TOPS  IN  SERVICE  CLASS 


ALPHA  (1,2,5,8,10,11,12,15)  8 
IRMX  (7,9,14)  3 
POSIX  (6,13,16)  3 
CRONUS  (4)  1 
MACH  (3)  1 
ARTX  0 
ORKID  0 
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7.5.2  Relative  Success  Ranking 

Table  7-8  plots  the  various  candidates'  relative  success  in  the  service 
class  evaluation.  It  shows  the  results  for  the  technical  services  classes,  2 
through  16.  Each  column  represents  one  candidate;  each  row,  from  one  to  seven, 
represents  the  relative  evaluation  position  (first,  second,  ...). 

The  top  row,  therefore,  shows  for  each  candidate  the  number  of  service  classes 
for  which  it  achieved  the  best  evaluation. 

This  chart  is  an  attempt  to  give  a  perspective  on  how  the  candidates  did 
in  the  service  class  evaluation  as  compared  to  each  other.  It  gives  no 
perspective  on  how  a  candidate  meets  the  service  class  requirements.  By  looking 
at  the  top  two  or  three  rows,  it  can  be  determined  if  some  candidates  did 
generally  better  than  others. 


TABLE  7-8.  "WIN,  PLACE,  SHOW,  ... 


ALPHA  ARTX  CRONUS  iRMX  MACH  ORKID  POSIX 


1st  7 

2nd  4 

3rd  0 

4th  4 

5th  0 

6th  0 

last  0 


0  13 

2  2  4 

1  2  5 

5  11 

4  11 

3  3  1 

0  5  0 


10  3 

10  2 

3  0  4 

112 
6  3  0 

13  4 

2  8  0 


7.5.3  Relative  Success  Ranking  with  Acceptance 

Another  view  of  the  data  is  the  chart  presented  in  Table  7-9.  Table  7-9 
was  used  to  better  understand  the  relationship  between  the  candidates,  * 
especially  the  three  (Alpha,  iRMX,  POSIX)  that  appear  to  be  technically 
superior.  This  shows  the  rank  ordering  of  all  candidates  for  service  classes  2- 
16,  along  with  their  scores.  When  attempting  to  make  sense  of  this  chart,  it  is 
useful  to  mark  boundaries  in  each  column.  Two  boundary  values  were  selected  at 
5  and  7.  It  was  deemed  that  a  value  of  7  or  above  showed  at  least  an  acceptable 
level  of  support  for  the  service  class.  A  value  of  less  than  5  indicated  an 
unacceptable  level  of  support  for  the  service  class.  The  results  are  summarized 
in  Table  7-10. 
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This  view  is  useful  when  determining  the  service  classes  which  are  not 
supported  by  a  certain  candidate.  From  this,  a  measure  of  the  'cost  to  fix'  can 
be  made  for  bringing  the  candidate  up  to  meet  the  OSSWG  requirements.  As  can  be 
seen  in  the  figure,  service  classes  10,  11,  12,  14,  and  16  have  no  candidates 
with  acceptable  scores.  Service  class  10  does  not  even  have  a  candidate  with  a 
moderate  score.  Service  class  4  has  none  of  the  final  three  in  the  acceptable 
or  moderate  range.  Table  7-10  shows  how  the  final  three  candidates  fared 
relative  to  the  5  and  7  boundaries. 

Some  conclusions  which  can  be  reached  from  the  service  class  analysis  are 
that  none  of  the  candidates  cover  a  wide  range  of  the  service  classes  really 
well.  Of  the  16  service  classes  Alpha  is  acceptable  in  5,  Posix  in  4,  and  iRMX 
in  3.  For  acceptable  plus  moderate  categories,  Alpha  hits  12  times,  Posix  8 
times,  and  iRMX  hits  7  times.  All  candidate  scores  were  low  in  service  classes 
10,  11,  12,  14  and  16.  This  may  indicate  a  lack  of  maturity  with  the  state-of- 
the- technology  with  which  no  system  is  yet  able  to  deal  successfully. 


TABLE  7-9.  SERVICE  CLASS  ANALYSIS 


Service  Class 


0 

1 

2 

3 

4 

5 

6 

7 

8 

P 

a 

a 

M 

C 

a 

P 

i 

a 

7.53 

7.56 

7.00 

7.31 

8.33 

7.38 

8.24 

8.40 

7.00 

i 

i 

C 

P 

a 

i 

i 

A 

C 

6.87 

7.19 

5.11 

7.28 

3.33 

5.42 

8.09 

7.57 

6.04 

A 

P 

i 

C 

M 

P 

A 

P 

i 

6.68 

6.46 

4.67 

6.48 

2.44 

3.88 

7.57 

7.12 

5.06 

C 

C 

A 

a 

P 

C 

a 

a 

A 

6.42 

6.39 

3.78 

1.74 

1.44 

3.71 

6.09 

5.85 

4.21 

M 

A 

M 

i 

A 

0 

M 

M 

M 

6.20 

5.91 

3.67 

0.95 

1.11 

2.56 

4.13 

1.58 

3.82 

0 

M 

P 

A 

i 

A 

C 

0 

P 

6.19 

5.65 

3.44 

0.56 

1.00 

2.30 

3.72 

0.58 

3.20 

G‘ 

0 

0 

0 

0 

M 

0 

C 

0 

6.01 

5.46 

2.44 

0.09 

0.78 

1.85 

0.00 

0.53 

0.50 

Note : 

a  -  ALPHA,  A  -  ARTX,  C  -  CRONUS,  i  =  iRMX,  M  -  MACH,  0  -  ORKID,  P  -  POSIX 
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TABLE  7-9  (Cont.) 


Service  Class:  9 

10 

11 

12 

13 

14 

15 

16 

i 

a 

a 

ct. 

P 

i 

a 

P 

8.09 

4.71 

5.50 

6.60 

8.11 

6.69 

7.71 

6.85 

a 

i 

i 

M 

a 

a 

P 

A 

7.99 

4.55 

4.16 

5.44 

6.94 

5.96 

6.86 

6.84 

M 

M 

C 

P 

i 

P 

i 

i 

7.53 

2.81 

1.81 

3.86 

6.31 

5.33 

4.88 

6.56 

A 

A 

P 

i 

A 

M 

0 

a 

6.56 

2.18 

1.78 

3.26 

5.23 

5.26 

3.69 

5.82 

0 

C 

M 

A 

0 

A 

A 

M 

6.26 

1.40 

0.84 

2.62 

5.16 

4.51 

2.96 

4.98 

P 

P 

A 

0 

M 

C 

C 

0 

6.21 

0.35 

0.21 

1.75 

3.79 

3.78 

2.17 

3.63 

C 

0 

0 

C 

C 

0 

M 

C 

3.92 

0.30 

0.19 

0.57 

3.38 

0.70 

1.79 

2.02 

Note : 

a  -  ALPHA,  A  -  ARTX, 

C  -  CRONUS,  i 

-  iRMX , 

M  -  MACH,  0  - 

ORKID, 

P  -  POSIX 
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TABLE  7-10.  COMPARISONS  AMONG  ALPHA,  IRMX,  AND  POSIX 


Acceptable 

Moderate 

Unacceptable 

2 

a 

i.  P 

3 

P 

a,  i 

4 

a,  i,  p 

5 

a 

i 

P 

6 

P.  i 

a 

7 

i,  P 

a 

8 

a 

i.  P 

9 

i,  a 

P 

10 

a,  i,  p 

11 

a 

i.  P 

12 

a 

P.  i 

13 

P 

a,  i 

14 

i,  a,  p 

15 

a 

P 

i 

16 

p,  i,  a 

Note: 

a  -  ALPHA,  i  -  iRMX,  p  -  POSIX 


7.6  VARIANCE  ANALYSIS 

There  often  appeared  to  be  a  significant  disagreement  between  the 
evaluators  with  regard  to  the  appropriate  score  for  a  candidate  on  a  given 
criterion.  This  can  be  seen  in  the  data  in  Appendix  E  where  the  difference 
between  the  "max"  and  "min"  columns  is  large,  usually  resulting  in 
correspondingly  large  sigma  (standard  deviation)  values.  Such  large  sigma 
values  indicate  that  the  variation  across  the  evaluator  scores  may  be  so  large 
(indicating,  in  statistical  terms,  a  large  error  in  the  sample)  that  we  would 
not  be  justified  in  placing  much  importance  on  an  apparently  significant 
difference  between  the  scores  of  two  candidates. 

It  is  believed  that  the  large  sigma  values  result  from  a  number  of 
circumstances,  among  them: 

o  different  interpretations  of  the  same  requirement  by  two  different 
evaluators 

o  different  evaluator  persistence  in  working  through  the  documentation 
to  determine  the  existence  or  non-existence  of  a  feature 

o  the  short  time  available  to  some  evaluators  on  some  candidates  due  to 
late  receipt  of  the  documentation 
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o  whether  or  not  a  given  evaluator  attended  the  January  OSSWG  meeting  in 
Mobile  at  which  the  candidates  were  briefed 

o  the  varying  nature  of  the  candidates  (all  the  way  from  one  designed  to 
be  incorporated  on  top  of  other  native  operating  systems  to  one 
intended  to  include  only  the  bare  necessities  of  a  real-time  kernel) 

o  misleading  cross-reference  matrices  from  some  candidates 

o  inconsistent  candidate  definitions  (i.e.,  some  evaluators  making  use 
of  some  documentation  which  other  evaluators  did  not  consult) 

o  the  difficulties  of  cross -application  of  documents;  e.g.,  in  the  case 
of  POSIX,  where  one  would  have  to  examine  both  1003.1  and  1003.4  in 
order  to  determine  a  proper  score  for  all  File  System  (Service  Class  6 
criteria) 

o  variations  in  evaluator  qualifications  for  evaluating  a  given  service 
class 

o  confusing  candidates/quality  of  documentation 

o  confusing  requirements  (criteria) 

o  the  difficulties  of  maintaining  the  distinction  between  "interface" 
and  "implementation"  semantics 

o  the  credibility  which  various  evaluators  were  willing  to  grant  to 
individual  candidates. 

One  way  in  which  these  large  sigmas  were  used  to  differentiate  between  the 
seven  candidates  was  to  examine  the  number  of  criterion  sigma  values  for  each 
candidate  which  were  greater  than  some  threshold  value.  This  was  intended  to 
give  some  indication  of  which  candidates  appeared  to  give  the  most  trouble  to 
the  evaluators.  For  the  threshold  value  we  chose  3.16,  which  is  the  value  that 
one  would  expect  for  the  standard  deviation  if  the  numbers  (scores)  were 
generated  randomly.  Using  the  sigmas  from  all  17  service  classes,  this  count 
yielded  the  results  in  Table  7-11. 
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TABLE  7-11.  SUMMARY  OF  ERROR  JESUITS  BY  CANDIDATE 


CANDIDATE 

#  CRITERIA  WITH 

SICMA  >  3.16 

(153  total  criteria) 

Mach 

80 

Cronus 

78 

Alpha 

70 

POSIX 

£2 

ARTX 

30 

iRMX 

ORKID 

22 

In  statistics,  the  question  of  sampling  error  is  dealt  witfc  ng-ing  the 
"analysis  of  variance."  The  formulae  applied  use  the  candidate  scores  and'£aeir 
corresponding  sigmas,  in  light  of  the  number  of  sampling  points  (in  our  case, 
number  of  evaluators  per  criterion) ,  to  when  one  can  say  with  high 

confidence  that  the  difference  between  two  or  more  sets  of  scores  is 
statistically  significant. 

Such  an  analysis  of  variance  was  first  applied  across  all  seven  candidates 
to  determine  whether  we  were  justified  in  saying  that  three  of  the  candidates 
(Alpha,  iRMX,  and  POSIX)  were,  as  a  group,  superior  to  the  four  -others..  It  was 
applied  using  all  17  service  classes.  The  hypothesis  was  ccmfiracri  by  the 
analysis,  showing  that  there  was  statistical  significance  to  the  wore 
differences  across  the  board,  despite  some  high  sigma  values.  This  analysis 
also  confirmed  that  the  evaluation  process  utilized  was  a  robust  one  resulting 
in  useful  and  usable  information. 

Having  confirmed  the  validity  of  concentrating  on  the  top  three,  we  mart 
needed  to  determine  ways  of  more  closely  examining  the  differences  between  them 
in  order  to  see  if  one  would  emerge  as  dominant  over  the  other  two.  However, 
first  it  was  necessary  to  apply  the  same  analysis  of  variance  to  the  top  three; 
if  none  of  the  score  differences  showed  theaselves  to  he  significant,  in  light 
of  the  sigmas,  then  we  would  not  find  grounds  in  the  scores  for  differentiating 
between  them. 

Thus,  an  analysis  of  variance  was  applied  -across  just  the  top  three 
candidates.  This  result  was  coupled  with  the  results  of  another  analysis  (which 
will  be  described  in  the  following  section)  to  hring  the  comparison  of  the  top 
three  down  to  a  manageable  and  reliable  set  of  information. 
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7.7  MAJOR  ISSUES  PARTITION  OF  REQUIREMENTS 

Although  the  RADs  did  not  reveal  what  had  been  expected  about  the 
candidates,  it  was  still  desirable  to  see  just  how  well  the  candidates  were 
likely  to  deal  with  certain  special  aspects  of  target  Navy  applications.  Early 
in  the  life  of  the  OSSWG  a  set  of  high-level  requirements  was  identified  from 
information  in  the  Operational  Requirements  and  the  Development  Options  Paper. 
These  requirements  were:  real-time,  distribution,  heterogeneity,  Ada,  security, 
and  reliability.  It  was  decided  that  a  useful  additional  view  of  the  data  would 
be  to  associate  various  requirements  (criteria)  with  these  six  high-level 
requirements  and  to  see  how  the  candidates  performed  with  respect  to  these  . 
selected  "slices"  through  the  Navy's  needs.  In  addition,  it  was  decided  to 
further  categorize  the  criteria  associated  with  each  major  issue  according  to 
whether  it  was  Essential  or  Non-Essential  (i.e.,  poor  performance  by  a  candidate 
on  an  Essential  criterion  would  be  considered  a  significant  black-mark  against 
that  candidate  and  possibly  grounds  for  finding  it  unacceptable) .  The  partition 
of  the  criteria  was  carried  out  by  members  of  the  Requirements  Subgroup.  The 
results  of  this  partitioning  can  be  viewed  in  Appendix  F,  Essential  Criteria 
Partitions . 

The  allocation  of  the  evaluation  criteria  to  the  high-level  requirements 
was  then  used  to  highlight  the  performance  of  the  top  three  candidates  on  those 
criteria  which  seemed  to  matter  most.  However,  in  order  to  do  so  justifiably, 
we  first  applied  the  results  of  the  variance  analysis  (see  Section  7.6),  which 
told  us  on  which  criteria  the  differences  between  the  top  three  candidates' 
scores  could  be  considered  statistically  significant.  This  reduced  the 
comparison  between  the  top  three  to  the  following  technical  criteria  (i.e.,  it 
does  not  address  the  programmatic) . 


TABLE  7-12.  ESSENTIAL  CRITERIA  FOR  WHICH  DIFFERENTIATION 
IN  TOP  THREE  CANDIDATES'  SCORES  IS  SIGNIFICANT 


Cr. 

Cr. 

Cr. 

Cr. 

Cr. 

Cr. 

Cr. 

1.4 

3.1 

3.9 

3.19 

11.5 

12.7 

15.7 

1.20 

3.2 

3.10 

3.22 

11.7 

12.8 

1.21 

3.3 

3.12 

5.4 

11.9 

12.9 

1.23 

3.4 

3.13 

5.5 

11.10 

12.10 

1.25 

3.5 

3.14 

8.2 

11.11 

13.5 

1.26 

3.6 

3.15 

8.9 

11.14 

13.9 

1.27 

3.7 

3.16 

9.13 

12.2 

15.1 

1.28 

3.7 

3.17 

10.2 

12.5 

15.4 

1.29 

3.8 

3.18 

11.4 

12.6 

15.5 

Finally,  a 

tally  was  made 

of  all  those  criteria  on 

the  above  list 

for  which 

each 

candidate  scored  below  5 

points;  a  second  tally  was 

made  of  all  those 

criteria  on  the 

list  for  which 

each  candidate  scored  7  or  above.  A  score  of  less 

than 

5  points  is 

considered  significant  because 

of  the  general  phrasing 

of  the 
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evaluation  criteria:  in  general,  a  score  of  5  is  used  to  describe  a  feature  as 
something  which  could  be  used  and  tolerated,  at  least  as  a  starting  point, 
whereas  a  score  lower  than  that  indicates  some  way  in  which  the  feature  is 
seriously  lacking  in  capability.  A  score  of  7  or  above  was  also  considered 

significant  because  it  indicates  a  level  .at  which  most. grading  scales  give . . 

passing  or  superior  assessments.  The  results  of  this  tallying  are  shown  in  - 
Table  7-13,  where  *  indicates  scores  above  7  and  -  indicates  scores  below  5. 

The  number  indicates  the  number  of  criteria  where  the  candidate  scored  less  than 
5  or  more  than  7 . 


TABLE  7-13.  RESULTS  OF  TALLY  CRITERIA  SCORES  <5  &  >7 
AMONG  ALPHA,  IRMX,  AND  POSIX 


SC 

ALPHA 

IRMX 

POSIX 

1 

8* 

3*  1- 

1*  3- 

3 

19- 

19- 

14* 

5 

1* 

2- 

2- 

8 

1* 

1*  1- 

2- 

9 

1* 

1- 

1- 

10 

1- 

1- 

1- 

11 

3- 

5- 

7- 

12 

2*  2- 

6- 

2*  5- 

13 

1* 

1- 

1*  1- 

15 

4* 

3- 

2*  2- 

TOTALS 

18*  25- 

4*  40- 

20*  24 

For  completeness,  when  a  similar  process  was  applied  to  the  programmatics 
(i.e.,  considering  only  those  criteria  for  which  the  variation  in  the  scores 
could  be  considered  significant;  note  that  the  programmatic  requirements  were 
not  allocated  to  the  six  high-level  requirements),  the  results  were  as  given  in 
Table  7-14  below. 


TABLE  7-14.  SIGNIFICANT  SCORES  FOR  CRITERIA  SCORES 
<5  &  >7  FOR  PROGRAMMATICS  AMONG  ALPHA,  IRMX,  AND  POSIX 


CRITERIA  ALPHA 


IRMX 


POSIX 


O.l 

0.3 

1* 

1* 

1* 

0.5 

1- 

1* 

0.7 

1* 

1* 

TOTAL 

1*  1- 

2* 

3* 
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CHAPTER  8 
CONCLUSIONS 


This  report  documented  the  results  of  the  OSSWG  evaluation  process,  carried 
out  in  order  to  select  a  baseline  candidate  operating  system  interface  for  NGCR 
standardization.  The  results  show  this  to  have  been  a  very  complete  look  at  the 
candidates.  They  also  give  reason  to  believe  that  the  process  as  conceived  and 
conducted  and  the  results  which  it  generated  were  as  good  as  could  possibly  have 
been  expected.  The  OSSWG  can  say  with  confidence  that  the  seven  candidates  were 
fairly  evaluated  and  that  they  can  reasonably  be  differentiated  into  two  groups: 
those  that  scored  in  the  top  three  positions  (Alpha,  iRMX,  and  POSIX)  and  the 
other  four  which  did  not. 
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APPENDIX  A 

DESCRIPTION  OF  CANDIDATES 


CANDIDATE:  Alpha 
DEFINING  DOCUMENTS: 

Alpha  Operating  System  Kernel  Interface  Specification, 

Revision  0.2,  June  1989. 

BACKGROUND  AND  STATUS 

The  Alpha  OS  arose  as  part  of  the  Archons  Project  on  new  paradigms  for 
real-time  decentralized  computer  systems,  which  began  at  Carnegie -Mellon 
University  (CMU)  in  1979.  Alpha  design  started  in  1985  and  the  initial 
prototype  was  operational  in  1987.  This  prototype  ran  on  multiprocessor  nodes 
built  by  modifying  Sun  workstations,  which  were  then  interconnected  by  an 
Ethernet.  Another  copy  of  this  Alpha  testbed  was  installed  at  General 
Dynamics'  Ft.  Worth  facility.  Alpha  was  sponsored  at  CMU  primarily  by  the 
USAF  Rome  Air  Development  Center  (RADC) ,  with  additional  funding  from  the 
Naval  Ocean  Systems  Center  (NOSC)  and  several  corporations.  The  focus  of 
Alpha  R&D  moved  in  1988  to  Concurrent  Computer  Corporation's  Boston  facility, 

.  where  it  continues  to  be  sponsored  in  part  by  RADC.  Concurrent  is  leading  a 
group  of  organizations  which  are  performing  a  second  generation,  commercial 
quality  design  and  implementation  of  the  Alpha  OS  in  a  series  of  increasing 
functionality  releases.  Release  2  of  the  kernel  is  scheduled  for  delivery  to 
a  number  of  Government  and  industry  sites  in  mid- 1990,  on  MIPS  based 
multiprocessors  interconnected  by  FDDI .  Alpha  is  portable  and  will  be  next 
available  on  68030/68040  multiprocessors;  Alpha  will  be  running  on  a 
multivendor  heterogeneous  RISC  and  CISC  system  by  the  end  of  1990.  Alpha- 
compatible  API's  will  be  available  for  UNIX  and  a  number  of  industry  real¬ 
time  executive  products,  and  a  POSIX-compliant  API  will  be  available  on  Alpha. 
Alpha  is  non-proprietary  and  in  the  public  domain  for  Government  use. 

INTENDED  TARGET  DOMAIN 

The  Alpha  OS  is  intended  for  larger,  more  complex,  more  distributed, 
mission-critical  real-time  systems.  The  most  demanding  such  systems  are 
always  found  first  in  military  warfare  environments ,  ranging  from  combat 
platform  and  battle  management  to  C3I,  due  to  the  conflicting  imperatives  of 
accommodating  the  large  number  of  extremely  dynamic  uncertainties  inherent  in 
hostile  missions,  while  nonetheless  assuring  maximal  dependability  of 
effectiveness,  survivability,  and  safety.  Alpha  is  already  being  designed 
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into  major  DoD  programs  for  land,  sea,  and  air  applications.  However,  similar 
needs  are  becoming  increasingly  common  in  industrial  (e.g.,  factory 
automation,  telecommunications),  and  even  in  commercial  (e.g.,  on-line 
transaction  processing),  applications.  Concurrent  is  committed  to  an  Alpha 
product  for  these  environments  as  well. 

TECHNICAL  OVERVIEW 

Alpha's  kernel  consists  of  mechanisms  which  are  directed  by  policy 
modules  at  the  system  layer  of  the  OS.  The  kernel  primitives  are  designed  to 
manage  all  physical  and  logical  resources  (such  as  processor  cycles,  memory, 
i/o,  semaphores,  atomic  transactions)  directly  with  the  actual  task  time 
constraints  and  relative  importance  specified  by  the  application.  Alpha's  API 
is  specifically  intended  for  distributed  real-time  programs  and  is  object 
oriented.  The  computational  entities  are  threads  which  span  objects  and 
(transparently  and  reliably)  physical  node  boundaries.  Threads  carry  their 
attributes  (such  as  real-time  and  transactional  state)  so  that  trans-node 
resource  manage,  ent  can  be  performed  in  the  best  interests  of  the  distributed 
applications  and  mission.  Alpha's  kernel  includes  mechanisms  for  real-time 
atomic  transactions  (non-serializable  as  well  as  serializable),  to  support 
maintaining  application-specific  correctness  of  distributed  execution  and 
consistency  of  distributed  (replicated  and  partitioned)  data.  One  unified 
exception  handling  construct  deals  with  unsatisfied  time  constraints, 
transaction  aborts,  hardware  faults,  and  application- specific  exceptions. 

REFERENCE 

Jensen,  E.  Douglas  and  J.  Duane  Northcutt,  "Alpha:  An  Open  Operating 
System  for  Mission  Critical  Real-Time  Distributed  Systems,"  Proceedings  1989 
Workshop  on  Operating  Systems  for  Mission-Critical  Computing,  ed.  Gordon, 
Hwang,  and  Agrawala,  ACM  Press,  1990. 


CANDIDATE:  ARTX  (Ada  Real-Time  executive) 

DEFINING  DOCUMENTS: 

VRTX32C,  Versatile  Real-Time  Executive,  User's  Guide 
MPV,  Multiprocessor  VRTX,  User's  Guide 
TNX-E,  TCP/IP  Network  Executive  Communication  Package, 
User's  GUIDE 
IFX  User's  Guide 

RTAda,  Real-Time  Ada,  User's  Guide 

RTAda,  Real-Time  Ada,  Board  Support  Package,  Developer's 
Guide  ARTX32  Engineering  Implementation 
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BACKGROUND 

ARTX  was  developed  by  Ready  Systems  to  implement  the  critical  "kernel" 
services  of  an  Ada  multitasking  real-time  Runtime  System  for  embedded 
microprocessor  applications.  It  is  upwardly  compatible  with  Ready  Systems' 
industry  standard  kernel  VRTX32,  so  that  application  tasks  wtitten  in  other 
languages  (C,  Fortran  and  assembly  language)  can  be  easily  integrated  into  the 
system  without  any  changes. 

INTENDED  TARGET  DOMAIN 

Ready  Systems'  family  of  operating  systems  has  been  targeted  for  two 
markets:  commercial  data  processing,  and  aerospace,  defense  and  engineering 
real-time  embedded  systems.  It  serves  as  the  basis  for  a  wide  range  of  diverse 
applications  such  as  accounting,  banking,  vehicle  control,  instrumentation, 
avionics,  and  telecommunications. 


TECHNICAL  OVERVIEW 

ARTX  supports  a  full  range  of  Ada  semantic  operations ,  including  the 
complete  Ada  tasking  model.  ARTX's  real-time  capabilities  include  the 
following:  deterministic  algorithms  with  fixed,  specified  timing  for  task 
rescheduling,  rendezvous  calls  and  accepts,  memory  allocation,  interrupt 
latency,  and  interrupts-off  time;  a  fully  preemptive  scheduler  so  that  the 
highest  priority  task  in  the  system  will  always  be  executing;  and  it  allows 
task  priority  to  be  changed  at  run- time.  ARTX  provides  additional 
communication  and  synchronization  primitives  besides  the  standard  Ada 
rendezvous.  ARTX  can  also  be  configured  to  provide  a  multi-processor 
•  networked  runtime  environment  by  using  its  two  companion  components: 
RTAda-MP,  which  supports  tightly-coupled,  shared  memory  multiprocessor 
systems,  and  RTAda-Net  for  multiprocessor  communication  over  local  area 
networks . 

REFERENCE 

"RTAda,  Real-Time  Ada,  User's  Guido,  for  Sun/Unix-to-68020, "  Ready 
Systems,  812-112-001,  Jun  1989 


CANDIDATE:  Cronus/SDOS  (Secure  Distributed  Operating  System) 
DEFINING  DOCUMENTS: 

Cronus  User's  Reference  Manual,  Release  1.4 
Cronus  Programmer's  Reference  Manual,  Release  1.4 
Cronus  Summary  Document  (cross-reference) 

Specification  Document  for  the  Experimental  Secure 
Distributed  Operating  System  Development,  6  Feb  1989 
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Design  Document  for  the  Experimental  Secure  Distributed 
Operating  System  Development,  2  Jun  1989 
Formal  Security  Model  for  the  Experimental  Secure 

Distributed  Operating  System  Development,  7  Oct  1989 
Software  Requirements  Specification  Documents  for  the 
Experimental  Secure  Distributed  Operating  System 
Development  (Volumes  I- -VI)  2  Jun  1989 
Software  Design  Document  for  the  Experimental  Secure  Distributed 
Operating  System  Development  (Volume  I- -Parti) 

Software  Design  Document  for  the  Experimental  Secure 
Distributed  Operating  System  Development 
(Volume  II  -  Parts  II--VIII),  30  Dec  1989  (Draft) 

BACKGROUND 

Cronus  has  been  under  development  at  BBN  Systems  and  Technologies 
Corporation  since  1981.  It  has  been  sponsored  by  the  Rome  Air  Development 
Center  (RADC)  to  address  the  problems  of  developing  and  maintaining  complex 
distributed  applications.  The  ultimate  goal  of  Cronus  is  to  integrate 
heterogeneous  computer  systems  into  an  effective  general-purpose  distributed 
computing  environment  for  the  development  and  execution  of  large-scale 
applications . 

SDOS  is  in  experimental  development  at  Odyssey  Research  Associates,  Inc, 
and  is  being  funded  by  RADC.  The  system  is  being  designed  and  built  to  meet 
TCSEC  B3  security  and  assurance  requirements.  SDOS  borrows  many  of  its 
concepts  from  Cronus,  but  the  system  architecture  has  been  redesigned  to 
provide  multi-level  security. 

INTENDED  TARGET  DOMAIN 

Cronus  is  an  environment  to  support  coherent  integration  of 
heterogeneous  (UNIX,  VMA,  other)  computer  systems.  Typically,  the  computer 
systems  fall  under  a  common  administrative  domain  and  are  interconnected  by 
one  or  more  high-speed  local  area  networks.  The  computer  systems  may  also  be 
interconnected  by  wide  area  networks,  via  an  internet  (such  as  the  DARPA 
Internet) . 

Cronus  is  currently  funded  by  the  government  for  both  system 
enhancements  and  for  use  in  Command  and  Control  applications  development. 
Cronus  is  presently  being  used  by  several  Navy  projects  such  as  the  Fleet 
Command  and  Control  Battle  Management  Program  (FCCBMP) . 

Command  and  Control  applications  potentially  needed  by  the  Air  Force  are 
representative  applications  that  SDOS  is  intended  to  support. 
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TECHNICAL  OVERVIEW 

Cronus  incorporates  many  desirable  distributed  processing 
features,  such  as  heterogeneity,  transparency  and  object-oriented 
programming,  as  well  as  high-level  features  such  as  survivability 
and  replication,  multi-site  database  access,  and  distributed 
monitoring  and  control  facilities. 

Cronus  is  designed  to  interoperate  with,  rather  than  to  replace  or 
totally  encapsulate,  constituent  (i.e.,  native)  operating  systems.  It  is  also 
designed  to  ease  the  integration  of  existing  software  into  a  distributed 
processing  environment.  It  provides  uniform  interprocess  communication 
portable  across  a  wide  range  of  host  and  operating  systems  types,  plus  an 
extensible  set  of  system  services  to  support  user  authentication,  symbolic 
naming,  configuration  management,  and  other  essential  functions.  It  also  has 
tools  to  facilitate  and  manage  the  development  of  complex  distributed 
applications  that  must  be  scalable,  survivable,  and  evolve  over  long 
lifetimes . 

SDOS  is  modeled  after  Cronus;  all  areas  are  the  same  except  security. 

It  has  a  layered  architecture,  built  on  top  of  a  secure  constituent  operating 
system  (UNIX  System  V/MLS) ,  which  will  provide  an  extensible,  small  Trusted 
Computing  Base  (TCB) . 

REFERENCES 

Vinter,  Stephen  T. ,  "Integrated  Distributed  Computing  Using 
Heterogeneous  Systems,"  SIGNAL.  £rmed  Forces  Communications  and 
Electronics  Association,  pp.  157-162,  Jun  1989 

Berets,  James  C. ,  Richard  M.  Sands,  "Introduction  to  Cronus,"  Technical 
Report  6986,  BBN  Systems  and  Technologies  Corporation,  Jan  1989 

Varadarajan,  R. ,  J.  R.  McEnerney,  and  D.G.  Weber,  "The  Secure 
Distributed  Operating  System  -  An  Overview,"  Proceedings  1989  Workshop  on 
Operating  Systems  for  Mission  Critical  Computing.  University  of  Maryland, 

Sept  1989.  ~~ 


CANDIDATE:  iRMX 
DEFINING  DOCUMENTS: 

iRMK  1.3  Real-Time  Kernel  Reference  Manual 
Distributed  iRMX  Nucleus  System  Calls 
Distributed  iRMX  I/O  System  Calls 
Distributed  iRMX  Networking  Services  System  Calls 
Distributed  iRMX  Application  Loader  System  Calls 
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Distributed  iRMX  Interactive  Configuration  Utility  User's 
Guide 

Distributed  iRMX  Configuration  Guide 
BACKGROUND  AND  STATUS 

The  Distributed  iRMX  Operating  System  was  developed  by  Intel  Corp.  to 
serve  as  the  latest  member  of  Intel's  iRMX  family  of  real-time  operating 
systems  which  has  been  used  in  approximately  7,500  designs  over  the  last 
decade.  The  iRMX  family  of  operating  systems  has  been  used  in  a  wide  range  of 
applications  including  avionics,  missile  control,  classified  Navy  projects, 
National  Security  Agency  projects,  radar  control  systems,  satellite 
communications,  and  a  host  of  non-military  applications.  The  Distributed  iRMX 
Operating  System  was  designed  from  scratch,  building  upon  the  extensive 
knowledge  obtained  from  the  development  of  earlier  members  of  the  iRMX  family 
of  operating  systems.  It  extends  the  capabilities  of  the  iRMX  operating 
systems  to  be  able  to  satisfy  the  needs  of  applications  of  much  greater 
complexity  that  were  currently  using  large  minicomputer  systems.  This  would 
be  accomplished  by  offering  a  distributed  operating  system  capable  of 
harnessing  the  computing  power  available  in  a  multicomputer  system  based  on 
386-based  single  board  computers  interconnected  by  the  MULTIBUS  II  bus. 

INTENDED  TARGET  DOMAIN 

The  iRMX  family  of  operating  systems  has  traditionally  been  sold  in  the 
OEM  market  with  the  operating  system  serving  as  the  basis  for  a  wide  variety 
of  applications  in  such  areas  as  military  systems,  process  control,  factory 
automation,  communications,  and  data  acquisition.  These  operating  systems 
have  been  used  in  applications  in  both  the  commercial  and  government  sector. 
Typical  applications  have  required  a  high  performance,  real-time  operating 
system.  The  Distributed  iRMX  Operating  System  will  be  sold  into  the  same  kind 
of  markets  but  will  provide  additional  computing  due  to  its  support  for 
distribution  and  fault- tolerance . 

TECHNICAL  OVERVIEW 

The  Distributed  iRMX  Operating  System  is  a  distributed,  real-time 
operating  system.  It  is  object  oriented,  with  a  core  set  of  objects  and 
corresponding  operations  defined  by  the  operating  system  along  with  facilities 
that  allow  the  user  to  define  additional  objects  and  operations  in  order  to 
extend  the  functionality  of  the  operating  system.  These  facilities  have  been 
designed  to  allow  transparent  access  to  objects  residing  on  remote  hosts.  In 
addition  to  facilities  to  support  transparent  distribution,  the  design 
includes  the  specification  of  additional  facilities  to  enable  fault- tolerant 
operation  of  the  system  to  ensure  that  tLe  distributed  operating  system 
continues  to  run  and  remain  consistent  in  the  presence  of  host  failures.  The 
facilities  to  support  fault- tolerance  include  the  ability  to  provide 
information  to  users  concerning  changes  in  system  configuration,  thus  enabling 
users  to  build  fault-tolerant  applications. 
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REFERENCE 

Saponas,  Timothy  G. ,  and  Demuth,  Roger  B. ,  "The  Distributed  iRMX 
Operating  System--A  Real-Time  Distributed  Operating  System,"  Proceedings  1989 
Workshop  on  Operating  Systems  for  Mission  Critical  Computing.  Karen  Gordon, 
Phil  Hwang,  and  Ashok  Agrawala  (ed.).  ACM  Press. 


CANDIDATE:  Mach 
DEFINING  DOCUMENTS: 

Mach  Kernel  Interface  Manual.  R.V.  Baron,  et  al.s  CMU, 

1  Jan  1990. 

"Network  Server  Design,"  Mach  Networking  Group, 

31  Aug  1989. 

"C  Threads,"  E.C.  Cooper  and  R.P.  Draves,  CMU,  20  Jul  1987. 

"Mach  Interface  Proposals- -Priorities ,  Handoff,  Wiring," 

D.L.  Black,  CMU,  14  Aug  1989. 

"The  Mach  cpu-server:  An  Implementation  of  Processor 
Allocation,"  D.L.  Black,  CMU,  14  Aug  1989. 

"Mach  Processor  Allocation  Interface,"  D.L.  Black,  CMU, 

14  Aug  1989. 

Mach  Programmer's  Manual:  Release  2.5+RTThread  [man  pages  on 
real-time  threads,  distributed  by  H.  Tokuda  at  OSSWG 
meeting  in  Jan  1990] . 

"MACH  Kernel  Modifications  for  the  Implementation  of  the 
Security  Policy,"  D.I.  Dalva,  Trusted  Information 
Systems,  Inc.,  Sept  1989. 

"Trusted  Mach  Name  Server  Interface  Document,"  H.  Taj  alii 
and  J.  Graham,  Trusted  Information  Systems,  Inc., 

15  Sept  1989. 

"Trusted  Mach  File  Server  Interface  Document,"  J.  Graham, 

Trusted  Information  Systems,  Inc.,  15  Sept  1989. 

"Trusted  Mach  Audit  Server  Interface  Document,"  J.  Graham, 

Trusted  Information  Systems,  Inc.,  19  Sept  1989. 

"Trusted  Mach  Verification  Server  Interface  Document,"  K.D. 

Henriksen  and  J.  Graham,  Trusted  Information  Systems, 

Inc. ,  15  Sept  1989. 

BACKGROUND 

The  Mach  project  was  initiated  at  Carnegie  Mellon  University  in  1984  as 
the  operating  system  effort  of  DARPA's  Strategic  Computing  Initiative  (SCI). 
DARPA  envisioned  Mach  as  a  vehicle  for  providing  a  uniform  (namely, 
UNIX-compatible)  software  base  across  the  architectures  existing  at  the  time, 
as  well  as  the  new  advanced  architectures  being  developed  as  part  of  the  SCI. 
Mach  has  lived  up  to  its  promise.  It  has  been  ported  to  numerous 
architectures,  system  platforms,  and  multiprocessors.  Commercial  versions  are 
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available  from  several  vendors,  including  BBN,  Encore,  and  NeXT.  The  Open 
Software  Foundation  (OSF)  has  incorporated  Mach  into  its  strategy  for 
achieving  a  general-purpose,  vendor-neutral  open  software  environment. 

INTENDED  TARGET  DOMAIN 

Mach  is  a  portable  minimal  kernel.  Various  operating  system  environments 
can  be  built  on  top  of  Mach.  However,  to  gain  leverage  from,  as  well  as  to 
offer  leverage  to,  the  abundant  supply  of  UNIX-based  software,  Mach  has  been 
purposefully  associated  with  UNIX  (namely,  4.3  BSD).  The  Mach  kernel  has 
traditionally  been  distributed  in  a  package  that  includes  an  operating  system 
environment  and  interface  offering  binary  compatibility  with  4.3  BSD.  In 
keeping  with  its  UNIX  heritage,  the  primary  application  domain  for  Mach  has 
been  interactive  computing.  But  through  DARPA- sponsored  projects  such  as 
Real-Time  Mach  and  Trusted  Mach,  Mach  is  being  extended  to  the 
mission-critical  computing  domain. 

TECHNICAL  OVERVIEW 

As  a  minimal  kernel,  Mach  implements  processes,  interprocess 
communication  (IPC),  and  memory  management.  Processes  are  implemented  through 
tasks  and  threads.  A  task  is  an  address  space  and  the  unit  of  resource 
allocation;  a  thread  is  the  unit  of  computation- -a  lightweight  process. 
Multiple  threads  can  execute  concurrently  within  a  task.  IPC  is  implemented 
through  ports  and  messages.  Ports  are  similar  to  capabilities,  in  terms  of 
their  roles  in  naming  and  protection.  Objects  are  represented  as  ports; 
operations  on  objects  are  performed,  subject  to  the  port  rights  of  the  sender, 
by  sending  messages  to  the  ports  that  represent  them.  Virtual  memory 
management  is  tightly  integrated  with  IPC,  and  it  is  distinguished  by  its 
portability,  advanced  functionality,  and  high  performance. 

REFERENCES 

Accetta,  M. ,  et  al.,  "Mach:  A  New  Kernel  Foundation  for  UNIX 
Development,"  Proceedings  Summer  USENIX.  Jul  1986 

Rashid,  R.F.,  et  al.,  "Mach:  A  Foundation  for  System  Software," 
Proceedings  1989  Workshop  on  Operating  Systems  for  Mission  Critical  Computing, 
ed.  Gordon,  Hwang,  and  Agrawala,  ACM  Press,  1990. 


CANDIDATE:  ORKID  (Open  Real-Time  Kernel  Interface  Definition) 
DEFINING  DOCUMENTS: 

ORKID:  Open  Real-Time  Kernel  Interface  Definition.  Draft  1.0 
for  Public  Comments,  July  1989. 
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BACKGROUND 

ORKID  is  being  developed  by  a  small  working  group  of  the  Software 
Subcommittee  o:.  the  VITA  Technical  Committee.  VITA  (VME  International  Trade 
Association)  is  an  organization  of  VMEbus  hardware  manufacturers.  VITA 
promotes  VMEbus  an  open  bus  architecture.  VITA  promotes  ORKID  as  an  open 
real-time  software  interface  that  is  not  particular  to  any  bus  or  hardware 
architecture.  The  ORKID  working  group  began  meeting  in  March  1988.  It  made 
ORKID  Draft  1.0  available  for  public  comments  in  July  1989.  The  ORKID  working 
group  reviews  comments  and  revises  the  draft  standard  at  quarterly  meetings. 

In  1990,  the  ORKID  working  group  plans  to  present  a  version  of  the  standard 
for  approval  to  the  VITA  membership  and  to  begin  marketing  it  to  other 
standards  organizations. 

INTENDED  TARGET  DOMAIN 

ORKID  specifies  an  application  program  interface  to  a  real-time  kernel. 
The  motivation  behind  its  development  is  to  allow  users  to  create  robust  and 
portable  code  (at  the  source  code  level) ,  while  also  allowing  implementors  the 
freedom  to  proliferate  their  compliant  products.  To  this  end,  ORKID  is 
designed  to  be  implementable  efficiently  across  a  wide  range  of 
microprocessors.  Furthermore,  ORKID  is  designed  to  operate  effectively  across 
a  wide  range  of  real-time  systems,  from  tightly  embedded  systems  to  complex 
multiprocessor  systems. 

TECHNICAL  OVERVIEW 

To  ensure  that  it  could  be  implemented  efficiently  and  also  to  enhance 
its  chances  of  acceptance  in  the  user  community,  ORKID  is  based  upon  proven 
technology.  The  ORKID  standard  defines  standard  kernel  object  types, 
including  standard  operations  on  those  types.  Standard  object  types  known  as 
tasks  and  queues  provide  for  processes  and  interprocess  communication. 

Standard  object  types  known  as  regions  and  partitions  provide  for  dynamic 
allocation  of  variable-sized  segments  and  fixed-sized  blocks,  respectively. 
Other  standard  object  types  include  semaphores,  events,  exceptions, 
interrupts,  a  clock,  and  timers.  At  the  present  time,  ORKID  does  not  provide 
for  virtual  memory,  but  the  ORKID  working  group  views  virtual  memory  support 
as  the  next  major  topic  on  its  agenda. 

REFERENCES 

The  defining  document  cited  above  is  the  best  reference. 


CANDIDATE:  POSIX  (IEEE  Standard  Portable  Operating  System  Interface  for 
Computer  Environments) 
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DEFINING  DOCUMENTS : 

1003.1-1988  IEEE  Standard  Portable  Operating  System 
Interface  for  Computer  Environments 

1003.2  (Draft)  Shell  and  Utilities 

1003.4  (Draft)  Realtime  Extensions 

1003.5  (Draft)  Ada  Language  Bindings 

1003.6  (Draft)  Trusted  System  Extensions 

1003.8  (Draft)  Networking  Standards 

BACKGROUND  AND  STATUS 

The  POSIX  standard  is  under  development  and  refinement  in  open  forum 
under  the  auspices  of  the  Computer  Society  of  the  Institute  of  Electrical  and 
Electronics  Engineers.  The  initial  standard  (1003.1-1988)  is  complete  and  has 
been  published;  the  remaining  portions  of  the  standard  defined  above  and 
reviewed  for  use  by  the  NGCR  OSSWG  are  either  currently  undergoing  the  IEEE 
balloting  process  (i.e.,  1003.2,  1003.4)  or  are  nearing  the  balloting  process. 
All  of  these  documents  are  expected  to  successfully  pass  balloting  before  the 
end  of  1991. 

INTENDED  TARGET  DOMAIN 

The  POSIX  standard  defines  a  robust  source-level  application  interface 
to  an  operating  system  providing  a  complete  set  of  services  supporting  the 
underlying  hardware.  It  is  addressed,  via  extensive  subsetting  options,  to  a 
wide  range  of  targets,  from  large,  distributed  application  domains  to  small, 
embedded  systems.  It  is  intended  that  secondary  standards  called  Application 
Environment  Profiles  will  define  the  specific  subset  definitions  for  specific 
application  domains. 

TECHNICAL  OVERVIEW 

Initially,  the  POSIX  standards  are  based  upon  interfaces  currently  in 
use  by  the  UNIX(tm-AT&T)  operating  system.  The  1003.1  interface  definition 
contains  the  basic  library  interfaces  to  support  processes  and  files  for  one 
or  more  simultaneous  users,  as  well  as  interfaces  to  commonly  supported 
devices  (e.g. ,  terminals  and  timers)  needed  by  applications.  The  1003.2 
standard  describes  the  interfaces  provided  by  a  "shell"  which  allows 
applications  to  be  managed  by  a  console  user,  as  well  as  interfaces  to  a 
number  of  commonly  used  utilities  (e.g.,  copying  files).  The  1003.4  standard 
defines  extensions  to  the  basic  1003.1  standard  to  support  applications  which 
need  predictable  time  response;  these  extensions  provide  interfaces  to 
alternate  functions  avoiding  unbounded  delays,  such  as  priority  scheduling, 
precision  clocks  and  timers,  asynchronous  I/O,  memory  locking,  contiguous 
files,  etc.  The  1003.5  standard  defines  Ada  interfaces  to  replace  the  C 
bindings  of  the  other  standards.  The  1003.6  standard  defines  the  requirements 
for  interfacing  to  c  trusted  (i.e.,  secure)  underlying  operating  system,  and 
the  1003.8  standard  defines  POSIX  application  interfaces  to  other  networking 
standards . 
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REFERENCE 

The  best  reference  to  the  scope  of  the  POSIX  work  is  found  in  the 
introduction  (and  Appendix  A)  to  the  1003.1-1988  standard  document  itself 
referenced  above.  The  document  is  available  from  the  IEEE  Press. 
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RANGE 


0.0-0. 4 


0.5-0. 9 


1.0-1. 4 


1.5-1. 9 


2. 0-2. 4 


2. 5-2. 9 


3. 0-3. 4 


3. 5-3. 9 


4. 0-4. 4 


4. 5-4. 9 


5. 0-5. 4 


5. 5-5.9 


6. 0-6. 4 


6. 5-6. 9 


7. 0-7. 4 


7. 5-7. 9 


8. 0-8. 4 


8. 5-8. 9 


9. 0-9. 4 


9. 5-9. 9 


TABLE  B-2.  DISTRIBUTION  OF  WEIGHTS 

OCCURRENCES 


X 


X 


XX 


XX 


X 


xxxxxx 


xxxxx 


xxxxxxxxxx 


xxxxxxxxxx 


xxxxxxxxxxxxxx 


xxxxxxxxx 


xxxxxxxxxxxxxx 


xxxxxxx 


xxxxxxxx 


xxxxxxxxxxx 


xxxxxxx 


xxxxxxx 


xxxx 


X 
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TABLE  B-3 .  RANGES  OF  WEIGHTS  FOR  EACH 
REPRESENTATIVE  APPLICATION  DOMAIN 


REPRESENTATIVE  APPLICATION  DOMAIN 

MIN  WEIGHT 

MAX  WEIGHT 

RUBY 

2.4 

9.2 

OPAL 

0.5 

7.3 

AMETHYST 

4.2 

9.5 

GARNET 

0.3 

8.5 

TOPAZ 

4.2 

8.8 

EMERALD 

1.7 

9.3 

DIAMOND 

1.7 

8.7 

SAPPHIRE 

3.4 

8.8 
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APPENDIX  C 

CONTRIBUTING  ORGANIZATIONS 


NAVY  ORGANIZATIONS 


NAVAL  AIR  DEVELOPMENT  CENTER 

NAVAL  UNDERWATER  SYSTEMS  CENTER 

NAVAL  OCEAN  SYSTEMS  CENTER 

NAVAL  SURFACE  WARFARE  CENTER 

NAVAL  WEAPONS  CENTER 

NAVAL  AVIONICS  CENTER 

NAVAL  RESEARCH  LABORATORY 

NAVAL  ORDNANCE  STATION 

NAVAL  AIR  TEST  CENTER 

SPACE  AND  NAVAL  WARFARE  SYSTEMS  COMMAND 

DEPARTMENT  OF  THE  NAVY  (INFORMATION  RESOURCES  MANAGEMENT) 

NAVAL  AIR  SYSTEMS  COMMAND 

Note:  NAVAL  SEA  SYSTEMS  COMMAND  (PMS-412)  participated  in 
the  evaluation  through  a  contractor. 


UNIVERSITY,  INDUSTRY,  AND  OTHER  GOVERNMENT  ORGANIZATIONS 

ARNOLD  ASSOCIATES 

BOOZ,  ALLEN,  &  HAMILTON  INC. 

CLEMSON  UNIVERSITY 
COMPUTER-BASED  SYSTEMS  INC. 

CONCURRENT  COMPUTER  CORP. 

DGM&S  INC. 

DIGITAL  EQUIPMENT  CORP. 

DYNAMICS  RESEARCH  CORP. 

ESL 

FORD  AEROSPACE 

GENERAL  ELECTRIC  AEROSPACE 

GENERAL  ELECTRIC  CO.  ADVANCED  TECHNOLOGY  LABORATORIES 
HONEYWELL  FEDERAL  SYSTEMS,  INC. 

IBM  CORP. 

INSTITUTE  FOR  DEFENSE  ANALYSES 
INTEL  CORP. 

IPI 

JOHNS  HOPKINS  UNIVERSITY/APPLIED  PHYSICS  LABORATORY 
LITTON  DATA  SYSTEMS 
MANAGEMENT  6c  ENGINEERING  INC. 
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MITRE  CORP. 

NATIONAL  INSTITUTE  OF  STANDARDS  AND  TECHNOLOGY 
OPEN  SOFTWARE  FOUNDATION 


UNIVERSITY,  INDUSTRY,  AND  OTHER  GOVERNMENT  ORGANIZATIONS 


PACIFIC  INTERNATIONAL  CENTER  FOR  HIGH  TECHNOLOGY  RESEARCH  (PICHTR) 
RAYTHEON  CO.  EQUIPMENT  DIVISION 
READY  SYSTEMS 

ROCKWELL  INTERNATIONAL  CORP. 

SCTC  INC. 

TEXAS  INSTRUMENTS,  INC. 

TRUSTED  INFORMATION  SYSTEMS 
TRW  FEDERAL  SYSTEMS  GROUP 
UNISYS  CORP. 
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APPENDIX  D 

FORMULAS  FOR  REDUCTION  AND  ANALYSIS  OF  EVALUATION 


D-l 


Formula*  for  Reduction  and  Analysis  of  Evaluation 


The  formulas  below  describe  the  processing  that  was  performed 
on  the  scores  and  confidence  values  collected  from  the 
evaluation  forms.  The  basic  operation  performed  on  the 
scores  was  the  aggregation  of  the  raw  scores  through  the  mean 
and  through  weighted  sums  to  produce  a  single  score  for  a 
service  class,  a  programmatic  issue  or  a  representative 
application  domain  (RAD) . 

There  are  two  estimates  of  the  errors  in  the  scores.  The 
first  estimate  is  the  standard  deviation  which  is  derived 
from  the  deviations  of  the  scores  submitted  by  different 
evaluators  for  the  same  criterion.  The  second  estimate  is 
from  the  confidence  value  that  the  evaluators  were  asked  to 
provide  for  each  score.  The  confidence  level  is  rated  as  "H" 
for  high,  "M"  for  medium,  or  "L"  for  low.  These  ratings  are 
converted  to  error  estimates  by  assuming  "H"  implies  a  low 
error  in  the  entered  score  (  ±  1)  while  the  "M"  implies  an 
error  estimate  of  +/-2  and  "L"  implies  a  larger  error  (  ± 

3) .  The  two  error  estimates  are  comparable  but  are  analyzed 
separately  to  allow  comparisons. 

Formulas : 


Reduction  on  Evaluators  yields  a  score  for  a  criterion: 


£(  ("ck  4)  2  +  (SC  sck)2)  (E<3«  5) 


I  XMC 


2.(  (wckro)2  +(so  sck)2)  <E9-  6> 


tapping  Service  Class  Scores  into  Representative  Application 
Domain  Scores: 


X'"km  *k  » 

service  classes 


X  Wkm 

service  classes 


(Eq.  7) 


X  »> 


service  classes 


X  (*km  4) 2  <E<*.  e> 


service  classes 


rm 


- irg-W  I  («k.  4) 2 

"km  yf  service  classes 
service  classes 


(Eq.  9) 
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Quantities  in  formulas: 

The  "raw"  score  for  candidate  i,  for  criterion  c 
from  evaluator  e. 

The  combined  score  for  for  candidate  i  for 
criterion  c 

The  number  of  evaluators  for  service  class  (or 
pragmatic  issue)  k. 

The  standard  deviation  for  the  score  for  candidate  i 
for  criterion  c. 

The  ("raw")  confidence  value  estimated  by 
evaluator  •  for  raw  score  S*#. 

The  estimated  confidence  value  for  S*. 

c 

The  weight  (0  . .  10  )  which  indicates  the 
importance  of  criterion  c  to  service  class  (or 
programmatic  issue)  k.  Where  0  indicates  the 
criterion  has  no  importance  to  this  service  class. 
These  weights  are  sometimes  referred  to  as  "weight 
set  1". 

The  standard  deviation  (error  estimate)  of  Wc)c. 

The  set  of  criteria  which  apply  to  service  class  k. 

The  Service  Class  score  for  service  class  (or 
pragmatic  issue)  k  for  candidate  i. 

i 

The  standard  deviation  for  8^. 

The  confidence  value  for  8^ 

The  weight  (0  ..  10  )  which  indicates  the  importance 
of  service  class  k  to  representative  application  m. 
These  weights  are  assumed  to  have  no  error.  These 
weights  are  sometimes  referred  to  as  "weight  set  2" 

The  score  for  representative  application  m  for 
candidate  i. 

The  standard  deviation  for  R^. 

The  confidence  value  for  R^. 
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APPENDIX  E 
CRITERIA  SCORES 

CANDIDATE:  ALPHA 

Results  with  data  for  evaluations  with  no  less  than  7  candidates  evaluated. 
221  files  for  candidate  ALPHA  for  Class  0  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

0 

1 

7.50000 

2.16430 

0.38079 

10 

5 

20 

0 

2 

6.10000 

2.53190 

0.44721 

10 

2 

20 

0 

3 

5.45000 

1.90498 

0.42426 

10 

2 

20 

0 

4 

6.75000 

2.04875 

0.40620 

10 

3 

20 

0 

5 

4.75000 

1.68195 

0.43301 

7 

0 

20 

0 

6 

6.70000 

2.25015 

0.41533 

10 

3 

20 

0 

7 

5.45000 

2.30503 

0.46098 

9 

0 

20 

0 

8 

5.40000 

2.92719 

0.51478 

10 

0 

20 

Mean  Service  Class  Score  for  Class  0: 

Score: 6. 01250  Sigma: 0.79741  Rho: 0.15449  evaluators : 20 . 00000 
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CANDIDATE:  ALPHA 


for  Class  1  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

1 

1 

6.86667 

2.01810 

0.26294 

10 

2 

45 

1 

2 

7.73333 

2.17841 

0.26667 

10 

0 

45 

1 

3 

6.08889 

2.04297 

0.26294 

10 

2 

45 

1 

4 

8.22222 

1.75666 

0.26010 

10 

4 

45 

1 

5 

8.11111 

2.45155 

0.25434 

10 

0 

45 

1 

6 

8.48889 

1.91433 

0.25240 

10 

1 

45 

1 

7 

8.55556 

1.94884 

0.25724 

10 

0 

45 

1 

8 

7.51111 

2.60787 

0.28458 

10 

0 

45 

1 

9 

7.24444 

2.59506 

0.27933 

10 

0 

45 

1 

10 

3.77778 

2.85155 

0.29481 

10 

0 

45 

1 

11 

6 . 64444 

2.94769 

0.26943 

10 

0 

45 

1 

12 

7.15556 

1.97663 

0.33921 

10 

2 

45 

1 

13 

7.86667 

1.73991 

0.30388 

10 

5 

45 

1 

14 

8.17778 

1.62773 

0.32811 

10 

5 

45 

1 

15 

7.42222 

2.00555 

0.28371 

10 

2 

45 

1 

16 

8.57778 

1.65816 

0.28631 

10 

5 

45 

1 

17 

8.80000 

1.47093 

0.31111 

10 

5 

45 

1 

18 

8.13333 

2.19089 

0.28371 

10 

0 

45 

1 

19 

7.08889 

2.86691 

0.31111 

10 

0 

45 

1 

20 

8.66667 

2.04495 

0.24845 

10 

0 

45 

1 

21 

7.64444 

2.78107 

0.25915 

10 

0 

45 

1 

22 

7.55556 

3.15188 

0.26200 

10 

0 

45 

1 

23 

8.35556 

2.84569 

0.27844 

10 

0 

45 

1 

24 

8.31111 

2.25451 

0.26943 

10 

2 

45 

1 

25 

9.42222 

0.98832 

0.25628 

10 

6 

45 

1 

26 

8.48889 

2.20147 

0.27487 

10 

2 

45 

1 

27 

7.55556 

2.34090 

0.28889 

10 

0 

45 

1 

28 

7.04444 

3.05224 

0.29979 

10 

0 

45 

1 

29 

6.62222 

2.51621 

0.29897 

10 

0 

45 

Mean  Service  Class  Score  for  Class  1: 


Score:7, 66282  Sigma:0. 72028  Rho:0. 57986  evaluators: 45. 00000 


for  Class  2  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

2  1  7.00000  3.04138  0.76980  10  0  9 

Weighted  Combined  Service  Class  Score  for  Class  2: 

Score: 7. 00000  Sigma: 3 .04138  Rho: 0.76980  evaluators : 9.00000 
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CANDIDATE:  ALPHA 


for  Class  3  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

3 

1 

1.42857 

1.81265 

0.57143 

5 

0 

7 

3 

2 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

3 

2.00000 

3.60555 

0.57143 

10 

0 

7 

3 

4 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

5 

2.57143 

2.93582 

0.57143 

8 

0 

7 

3 

6 

2.28571 

2.75162 

0.57143 

8 

0 

7 

3 

7 

2.00000 

1.63299 

0.57143 

5 

0 

7 

3 

8 

3.00000 

3.36650 

0.57143 

10 

0 

7 

3 

9 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

10 

1.71429 

1.79947 

0.57143 

5 

0 

7 

3 

11 

2.00000 

3.60555 

0.57143 

10 

0 

7 

3 

12 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

13 

1.85714 

1.77281 

0.57143 

5 

0 

7 

3 

14 

1.71429 

1.79947 

0.57143 

5 

0 

7 

3 

15 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

16 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

17 

2.28571 

1.79947 

0.57143 

5 

0 

7 

3 

18 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

19 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

20 

2.00000 

3.60555 

0.57143 

10 

0 

7 

3 

21 

2.85714 

3.76070 

0.57143 

10 

0 

7 

3 

22 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

23 

1.28571 

1.79947 

0.57143 

5 

0 

7 

3 

24 

2.00000 

3.60555 

0.57143 

10 

0 

7 

Weighted  Combined  Service  Class  Score  for  Class  3: 

Score: 1,74265  Sigma: 0.53711  Rho: 0.12758  evaluators: 7. 00000 


for  Class  4  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

4  1  3.33333  3.67423  0.57735  10  0  9 

Weighted  Combined  Service  Class  Score  for  Class  4: 

Score: 3. 33333  Sigma: 3. 67423  Rho: 0.57735  evaluators: 9. 00000 


for  Class  5  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

5 

1 

8.53846 

2.06621 

0.46790 

10 

5 

13 

5 

2 

7.23077 

2.45472 

0.46790 

10 

3 

13 

5 

3 

7.69231 

2.65784 

0.46790 

10 

3 

13 

5 

4 

6.69231 

3.14602 

0.52736 

10 

0 

13 

5 

5 

7.00000 

3.26599 

0.57048 

10 

1 

13 

5 

6 

7.00000 

3.26599 

0.57048 

10 

1 

13 

Weighted  Combined  Service  Class  Score  for  Class  5: 

Score: 7. 37543  Sigma: 1.59842  Rho: 1.11909  evaluators: 13. 00000 
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CANDIDATE:  ALPHA 


for  Class  6  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max  Min 

#Evaluators 

6 

1 

6.62500 

4.24054 

0.82916 

10 

0 

8 

6 

2 

5.87500 

4,48609 

0.82916 

10 

0 

8 

6 

3 

6.87500 

4.58063 

0.80039 

10 

0 

8 

6 

4 

7.37500 

4.56501 

0.75000 

10 

0 

8 

6 

5 

5.75000 

4.16619 

0 . 78062 

10 

0 

8 

6 

6 

5.25000 

3.84522 

0.75000 

10 

0 

8 

6 

7 

5.25000 

4.68280 

0.72887 

10 

0 

8 

6 

8 

5.25000 

4.68280 

0.72887 

10 

0 

8 

6 

9 

5.75000 

4.86239 

6.59948 

10 

0 

8 

6 

10 

5.87500 

4.96955 

0.59948 

10 

0 

8 

6 

11 

5.87500 

4.96955 

0.59948 

10 

0 

8 

6 

12 

6.50000 

4.40779 

0.71807 

10 

0 

8 

6 

13 

6.50000 

4.40779 

0.71807 

10 

0 

8 

6 

14 

6.50000 

4.40779 

0.71807 

10 

0 

8 

6 

15 

5.25000 

4.30116 

0.63738 

10 

0 

8 

6 

16 

6.00000 

5.01427 

0.59948 

10 

0 

8 

6 

17 

7.25000 

4.52769 

0.59948 

10 

0 

8 

6 

18 

7.25000 

4.52769 

0.63738 

10 

0 

8 

6 

19 

5.37500 

4.74906 

0.59948 

10 

0 

8 

6 

20 

5.50000 

4.98569 

0.78062 

10 

0 

8 

Weighted  Combined  Service  Class 

Score  for 

Class 

6: 

Score: 6. 09314  Sigma: 1.14066  Rho: 0.47735  evaluators : 8 . 00000 


for  Class  7  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max  Min 

#Evaluators 

7 

1 

6.50000 

3.11677 

0.85696 

10 

2 

8 

7 

2 

5.62500 

3.37797 

0.90139 

10 

1 

8 

7 

3 

5.62500 

3.37797 

0.90139 

10 

1 

8 

7 

4 

5.62500 

3.37797 

0.90139 

10 

1 

8 

7 

5 

5.62500 

3.37797 

0.90139 

10 

1 

8 

7 

6 

7.50000 

2.00000 

0.85696 

10 

5 

8 

7 

7 

5.62500 

3.37797 

0.90139 

10 

1 

8 

7 

8 

5.37500 

3.24863 

0.90139 

10 

1 

8 

7 

9 

6.87500 

3.52288 

0.87500 

10 

2 

8 

7 

10 

4.50000 

2.92770 

0.85696 

10 

1 

8 

7 

11 

4.87500 

2.74838 

0.90139 

10 

1 

8 

Weighted  Combined  Service  Class 

Score  for  Class 

7: 

Score: 

: 5. 85028  Sigma: 

1.08954  Rho 

:0. 56273 

evaluators : 

:  8.00000 
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CANDIDATE:  ALPHA 


for  Class  8  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

8 

1 

8.09091 

2.34327 

0.53009 

10 

2 

11 

8 

2 

8.27273 

2.53341 

0.53009 

10 

2 

11 

8 

3 

6.81818 

3.18805 

0.60984 

10 

0 

11 

8 

4 

6.72727 

3.19659 

0.60984 

10 

0 

11 

8 

5 

6.36364 

3.77552 

0.58916 

10 

0 

11 

8 

6 

7.27273 

3.25856 

0.60984 

10 

0 

11 

8 

7 

6.90909 

2.80908 

0.64282 

10 

2 

11 

8 

8 

6.54545 

3.04512 

0.62984 

10 

0 

11 

8 

9 

6.27273 

3.31936 

0.62984 

10 

0 

11 

8 

10 

5.72727 

3.77070 

0.68030 

10 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  8: 

Score: 7. 00022  Sigma: 1.38290  Rho: 0.99275  evaluators: 11. 00000 


for  Class  9  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max  Min 

#Evaluators 

9 

1 

8.55556 

2.29734 

0.50918 

10 

3 

9 

9 

2 

8.88889 

2.26078 

0.50918 

10 

3 

9 

9 

3 

8.44444 

2.40370 

0.56656 

10 

3 

9 

9 

4 

6.66667 

2.54951 

0.64788 

10 

3 

9 

9 

5 

8.77778 

2.27913 

0.54433 

10 

3 

9 

9 

6 

9.00000 

2.29129 

0.50918 

10 

3 

9 

9 

7 

7.55556 

3.12694 

0.47140 

10 

1 

9 

9 

8 

6.11111 

3.44400 

0.54433 

10 

0 

9 

9 

9 

8.00000 

2.54951 

0.56656 

10 

3 

9 

9 

10 

7.00000 

2.64575 

0.59835 

10 

3 

9 

9 

11 

7.88889 

2.52212 

0.64788 

10 

3 

9 

9 

12 

8.44444 

2.45515 

0.56656 

10 

3 

9 

9 

13 

8.11111 

1.96497 

0.56656 

10 

5 

9 

9 

14 

8.11111 

2.75882 

0.54433 

10 

3 

9 

Weighted  Combined  Service  Class 

Score  for 

Class 

9: 

Score: 7. 99199  Sigma: 1.02226  Rho: 0.76154  evaluators: 9. 00000 


for  Class  10  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

10 

1 

5.70000 

4.08384 

0.50000 

10 

0 

10 

10 

2 

3.70000 

2.49666 

0.60000 

8 

0 

10 

Weighted  Combined  Service  Class  Score  for  Class  10: 

Score: 4. 71191  Sigma: 3 .02544  Rho: 1.87466  evaluators: 10. 00000 
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CANDIDATE:  ALPHA 


for  Class  11  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

11 

1 

6.58333 

4.14418 

0.47871 

10 

0 

12 

11 

2 

6.41667 

3.70401 

0.53359 

10 

0 

12 

11 

3 

4.66667 

3.14305 

0.50000 

10 

0 

12 

11 

4 

4.50000 

3.14787 

0.47871 

10 

0 

12 

11 

5 

4.58333 

2.57464 

0.47871 

8 

0 

12 

11 

6 

3.91667 

3.47611 

0.53359 

10 

0 

12 

11 

7 

6.16667 

3.58870 

0.50000 

10 

0 

12 

11 

8 

4.41667 

3.60450 

0.53359 

10 

0 

12 

11 

9 

5.91667 

3.57919 

0.47871 

10 

0 

12 

11 

10 

6.50000 

3.96576 

0.54645 

10 

0 

12 

11 

11 

4.66667 

3.55050 

0.51370 

10 

0 

12 

11 

12 

5.58333 

3.31548 

0.57735 

10 

0 

12 

11 

13 

7.25000 

2.34036 

0.66667 

10 

4 

12 

11 

14 

5.58333 

4.01040 

0.54645 

10 

0 

12 

Weighted  Combined  Service  Class  Score  for  Class  11: 

Score: 5. 50050  Sigma: 1.12367  Rho: 0.62468  evaluators: 12. 00000 


for 

Class 

12  Mean  scores  for 

each  Criterion 

are-: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Eva!uators 

12 

1 

7.45455 

3.11010 

0.61658 

10 

2 

11 

12 

2 

7.36364 

3.07482 

0.69234 

10 

2 

11 

12 

3 

9.45455 

0.82020 

0.57496 

10 

8 

11 

12 

4 

6.54545 

3.90803 

0.62984 

10 

0 

11 

12 

5 

8.36364 

1.96330 

0.59613 

10 

5 

11 

12 

6 

5.72727 

3.46672 

0.61658 

10 

0 

11 

12 

7 

4.63636 

3.90571 

0.59613 

10 

0 

11 

12 

8 

4.63636 

3.90571 

0.59613 

10 

0 

11 

12 

9 

5.72727 

3.43776 

0.58210 

10 

0 

11 

12 

10 

5.63636 

3.44304 

0.56040 

10 

0 

11 

12 

11 

7.18182 

3.70994 

0.62984 

10 

0 

11 

12 

12 

5.09091 

2.84445 

0.64922 

10 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  12: 

Score: 6. 59533  Sigma: 1.15673  Rho: 0.72626  evaluators: 11. 00000 
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CANDIDATE:  ALPHA 


for  Class  13  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev 

13  1  8.36364  1.96330 

13  2  7.90909  2.87939 

13  3  4.90909  3.11302 

13  4  6.36364  3.61311 

13  5  8.09091  2.34327 

13  6  6.90909  3.33030 

13  7  7.18182  2.48267 

13  8  7.54545  2.50454 

13  9  5.45455  1.50756 

13  10  7.09091  2.58668 

13  11  5.27273  2.61116 

Weighted  Combined  Service  Class  Score  for  Class  13: 

Score: 6. 93892  Sigma: 1.07270  Rho: 0.70945  evaluators: 11. 00000 


for  Class  14  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

14 

1 

6.33333 

4.09268 

0.65734 

10 

0 

9 

14 

2 

6.77778 

4.02423 

0.65734 

10 

0 

9 

14 

3 

4.44444 

4.18662 

0.72860 

10 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  14: 

Score: 5. 95858  Sigma: 2. 53252  Rho: 0.96473  evaluators: 9.00000 


Rho 

Max 

Min 

#Evaluators 

0.48105 

10 

5 

11 

0.50616 

10 

2 

11 

0.56773 

10 

0 

11 

0.56773 

10 

0 

11 

0.48105 

10 

5 

11 

0.53009 

10 

0 

11 

0.53009 

10 

4 

11 

0.54545 

10 

5 

11 

0.58210 

10 

5 

11 

0.58210 

10 

3 

11 

0.56773 

9 

0 

11 

for  Class  15  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

15 

1 

8.69231 

2.17503 

0.44189 

10 

5 

13 

15 

2 

8.30769 

3.11942 

0.42829 

10 

0 

13 

15 

3 

4.84615 

3.78255 

0.56527 

10 

0 

13 

15 

4 

7.92308 

3.45112 

0.51025 

10 

0 

13 

15 

5 

8.00000 

3.39116 

0.52172 

10 

0 

13 

15 

6 

8.61538 

2.78503 

0.46154 

10 

0 

13 

15 

7 

7.00000 

4.00000 

0.44189 

10 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  15: 

Score: 7. 71179  Sigma: 1.56847  Rho: 1.01730  evaluators: 13. 00000 
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CANDIDATE:  ALPHA 


for 

Class 

16  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

16 

1 

6.44444 

2.45515 

0.70273 

10 

3 

9 

16 

2 

6.11111 

2.08833 

0.70273 

10 

3 

9 

16 

3 

6.77778 

2.72845 

0.67586 

10 

3 

9 

16 

4 

6.77778 

2.72845 

0.67586 

10 

3 

9 

16 

5 

5.66667 

1.65831 

0.70273 

8 

3 

9 

16 

6 

6.11111 

2.14735 

0.70273 

10 

3 

9 

16 

7 

4.77778 

2.27913 

0.67586 

8 

0 

9 

16 

8 

4.77778 

2.27913 

0.67586 

8 

0 

9 

16 

9 

6 . 44444 

2.24227 

0.67586 

10 

5 

9 

16 

10 

5.88889 

1.83333 

0.67586 

10 

5 

9 

16 

11 

5.77778 

3.03223 

0.67586 

10 

1 

9 

16 

12 

6.77778 

2.72845 

0.67586 

10 

3 

9 

16 

13 

6.22222 

2.48886 

0.72008 

10 

3 

9 

16 

14 

6.44444 

2.24227 

0.67586 

10 

5 

9 

16 

15 

5.33333 

2.69258 

0.67586 

10 

0 

9 

16 

16 

5.33333 

2.69258 

0.67586 

10 

0 

9 

16 

17 

5.11111 

1.26930 

0.67586 

8 

3 

9 

16 

18 

5.11111 

1.26930 

0.67586 

8 

3 

9 

16 

19 

5.88889 

2.84800 

0.67586 

10 

0 

9 

16 

20 

5.00000 

2.17945 

0.67586 

8 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  16: 

Score: 5. 82425  Sigma: 0.75630  Rho: 0.54562  evaluators : 9 . 00000 
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CANDIDATE:  ARTX 


Results  with  data  for  evaluations  with  no  less  than  7  candidates  evaluated. 
221  files  for  candidate  ARTX 

for  Class  0  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

0 

1 

5.45000 

1.27630 

0.41833 

10 

4 

20 

0 

2 

5.10000 

2.19809 

0.51720 

9 

1 

20 

0 

3 

8.15000 

1.84320 

0.41533 

10 

5 

20 

0 

4 

8.00000 

1.74718 

0.45552 

10 

5 

20 

0 

5 

6.20000 

2.85804 

0.46904 

10 

0 

20 

0 

6 

8.25000 

1.83174 

0.43875 

10 

5 

20 

0 

7 

6.45000 

1.46808 

0.46904 

9 

5 

20 

0 

8 

5.80000 

2.58742 

0.51720 

10 

0 

20 

Mean  Service  Class  Score  for  Class  0: 

Score: 6. 67500  Sigma: 0.72116  Rho: 0.16406  evaluators: 20. 00000 
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CANDIDATE:  ARTX 


for  Class  1  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

1 

1 

5.97778 

1.80264 

0.27035 

10 

2 

45 

1 

2 

6.84444 

2.31552 

0.28545 

10 

2 

45 

1 

3 

6.75556 

2.03554 

0.29481 

10 

2 

45 

1 

4 

6.88889 

2.49747 

0.29144 

10 

0 

45 

1 

5 

7.71111 

1.98428 

0.28974 

10 

2 

45 

1 

6 

7.22222 

2.52162 

0.28458 

10 

0 

45 

1 

7 

6.95556 

2.21519 

0.30551 

10 

1 

45 

1 

8 

6.53333 

2.22179 

0.29731 

10 

0 

45 

1 

9 

4.84444 

3.30259 

0.28889 

10 

0 

45 

1 

10 

7.82222 

2.78216 

0.26667 

10 

0 

45 

1 

11 

7.28889 

2.60787 

0.27666 

10 

1 

45 

1 

12 

7.26667 

1.87568 

0.30062 

10 

3 

45 

1 

13 

7.40000 

2.17841 

0.30062 

10 

0 

45 

1 

14 

6.97778 

2.62409 

0.31348 

10 

0 

45 

1 

15 

6.95556 

1.96510 

0.30144 

10 

3 

45 

1 

16 

6.97778 

2.14782 

0.32584 

10 

0 

45 

1 

17 

5.71111 

2.58160 

0.30225 

10 

0 

45 

1 

18 

7.20000 

1.79139 

0.29897 

10 

5 

45 

1 

19 

4.48889 

3.13066 

0.32508 

10 

0 

45 

1 

20 

6.97778 

2.40727 

0.29979 

10 

0 

45 

1 

21 

6.33333 

2.51360 

0.28974 

10 

0 

45 

1 

22 

7.46667 

1.94936 

0.28717 

10 

2 

45 

1 

23 

4.40000 

2.82360 

0.29481 

10 

0 

45 

1 

24 

1.04444 

1.95350 

0.28889 

10 

0 

45 

1 

25 

5.04444 

2.51320 

0.31348 

10 

0 

45 

1 

26 

4.24444 

2.89322 

0.32508 

10 

0 

45 

1 

27 

1.95556 

2.63676 

0.32508 

10 

0 

45 

1 

28 

5.00000 

2.45875 

0.32735 

10 

0 

45 

1 

29 

2.28889 

2.93585 

0.28889 

10 

0 

45 

Mean  Service  Class  Score  for  Class  1: 

Score: 5. 90738  Sigma: 0.65184  Rho:0. 46882  evaluators: 45. 00000 


for  Class  2  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

2  1  3.77778  2.16667  0.68493  609 

Weighted  Combined  Service  Class  Score  for  Class  2: 

Score: 3. 77778  Sigma: 2. 16667  Rho: 0.68493  evaluators: 9. 00000 
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CANDIDATE:  ARTX 


for  Class  3  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

3 

1 

0.71429 

1.88982 

0.51508 

5 

0 

7 

3 

2 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

3 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

4 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

5 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

6 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

7 

1.85714 

3.76070 

0.51508 

10 

0 

7 

3 

8 

1.85714 

3.76070 

0.51508 

10 

0 

7 

3 

9 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

10 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

11 

0.71429 

1.88982 

0.51508 

5 

0 

7 

3 

12 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

13 

0.42857 

1.13389 

0.57143 

3 

0 

7 

3 

14 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

15 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

16 

0.28571 

0.75593 

0.51508 

2 

0 

7 

3 

17 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

18 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

19 

0.42857 

1.13389 

0.51508 

3 

0 

7 

3 

20 

0.71429 

1.88982 

0.57143 

5 

0 

7 

3 

21 

0.71429 

1.88982 

0.57143 

5 

0 

7 

3 

22 

0.42857 

1.13389 

0.57143 

3 

0 

7 

3 

23 

0.42857 

1.13389 

0.57143 

3 

0 

7 

3 

24 

0.71429 

1.88982 

0.57143 

5 

0 

7 

Weighted  Combined  Service  Class  Score  for  Class  3: 

Score: 0.55882  Sigma :0. 33008  Rho: 0.11751  evaluators: 7. 00000 


for  Class  4  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

4  1  1.11111  3.33333  0.54433  10  0  9 

Weighted  Combined  Service  Class  Score  for  Class  4: 

Score: 1.11111  Sigma: 3. 33333  Rho: 0.54433  evaluators: 9. 00000 


for  Class  5  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

5 

1 

3.46154 

1.71345 

0.56001 

5 

0 

13 

5 

2 

2.38462 

2.32875 

0.54393 

6 

0 

13 

5 

3 

2.23077 

1.87767 

0.54393 

5 

0 

13 

5 

4 

0.61538 

1.19293 

0.49852 

3 

0 

13 

5 

5 

2.92308 

2.62874 

0.48650 

8 

0 

13 

5 

6 

2.00000 

2.41523 

0.48650 

7 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  5: 

Score: 2. 30391  Sigma :0. 94164  Rho: 0.42565  evaluators: 13. 00000 
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NAVSWC  TR  90-246 


CANDIDATE:  ARTX 


for  Class  6  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

6 

1 

6.75000 

3.49489 

0.75000 

10 

0 

8 

6 

2 

3.87500 

2.41646 

0.78062 

6 

0 

8 

6 

3 

2.75000 

2.71241 

0.85696 

7 

0 

8 

6 

4 

7.37500 

3.46152 

0.78062 

10 

0 

8 

6 

5 

5.50000 

3.81725 

0.72887 

10 

0 

8 

6 

6 

1.37500 

1.84681 

0.72887 

5 

0 

8 

6 

7 

9.37500 

0.91613 

0.67315 

10 

8 

8 

6 

8 

9.62500 

0.74402 

0.69597 

10 

8 

8 

6 

9 

9.25000 

1.16496 

0.67315 

10 

7 

8 

6 

10 

8.37500 

3.46152 

0.63738 

10 

0 

8 

6 

11 

9.75000 

0.70711 

0.59948 

10 

8 

8 

6 

12 

9.62500 

1.06066 

0.63738 

10 

7 

8 

6 

13 

10.00000 

0.00000 

0.71807 

10 

10 

8 

6 

14 

10.00000 

0.00000 

0.55902 

10 

10 

8 

6 

15 

0.00000 

0.00000 

0.59948 

0 

0 

8 

6 

16 

9.37500 

1.76777 

0.59948 

10 

5 

8 

6 

17 

8.75000 

2.12132 

0.63738 

10 

4 

8 

6 

18 

8.75000 

2.12132 

0.69597 

10 

4 

8 

6 

19 

8.12500 

2.23207 

0.72887 

10 

5 

8 

6 

20 

5.75000 

3.45378 

0.87500 

10 

0 

8 

Weighted  Combined  Service  Class  Score  for  Class  6: 

Score: 7. 57321  Sigma: 0.72398  Rho: 0.55498  evaluators: 8. 00000 


for  Class  7  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

7 

1 

9.00000 

1.19523 

0.59948 

10 

7 

8 

7 

2 

9.12500 

1.12599 

0.63738 

10 

7 

8 

7 

3 

9.12500 

1.24642 

0.69597 

10 

7 

8 

7 

4 

8.25000 

2.71241 

0.69597 

10 

2 

8 

7 

5 

8.25000 

2.71241 

0.69597 

10 

2 

8 

7 

6 

4.37500 

3.46152 

0.72887 

10 

0 

8 

7 

7 

8.75000 

1.58114 

0.59948 

10 

6 

8 

7 

8 

6.25000 

2.25198 

0.76035 

10 

3 

8 

7 

9 

6.00000 

3.74166 

0.66144 

10 

0 

8 

7 

10 

7.62500 

2.97309 

0.69597 

10 

1 

8 

7 

11 

5.12500 

2.90012 

0.76035 

10 

1 

8 

Weighted  Combined  Service  Class  Score  for  Class  7: 

Score:7. 57396  Sigma:0, 96979  Rho:0. 66297  evaluators : 8 .00000 
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CANDIDATE:  ARTX 


for  Class  8  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

8 

1 

2.72727 

2.14900 

0.55298 

5 

0 

11 

8 

2 

4.54545 

4.03395 

0.53009 

10 

0 

11 

8 

3 

6.45455 

3.61562 

0.48956 

10 

0 

11 

8 

4 

4.18182 

3.48764 

0.59613 

10 

0 

11 

8 

5 

2.81818 

3.31114 

0.60984 

7 

0 

11 

8 

6 

7.36364 

3.17089 

0.53009 

10 

0 

11 

8 

7 

5.45455 

4.00908 

0.51426 

10 

0 

11 

8 

8 

4.00000 

3.00000 

0.55298 

10 

0 

11 

8 

9 

2.27273 

2.93567 

0.53009 

8 

0 

11 

8 

10 

1.09091 

2.02260 

0.53009 

5 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  8: 

Score: 4. 21190  Sigma: 1.22583  Rho: 0.65922  evaluators: 11.00000 


for  Class  9  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

9 

1 

7.77778 

1.85592 

0.50918 

10 

5 

9 

9 

2 

6.55556 

2.24227 

0.50918 

10 

3 

9 

9 

3 

6.00000 

2.91548 

0.50918 

10 

0 

9 

9 

4 

4.00000 

3.39116 

0.50918 

9 

0 

9 

9 

5 

8.77778 

1.64148 

0.47140 

10 

5 

9 

9 

6 

8.77778 

1.64148 

0.47140 

10 

5 

9 

9 

7 

6.11111 

3.10018 

0.50918 

10 

0 

9 

9 

8 

8.00000 

1.41421 

0.50918 

10 

6 

9 

9 

9 

7.22222 

2.53859 

0.50918 

10 

3 

9 

9 

10 

4.33333 

2.78388 

0.50918 

9 

0 

9 

9 

11 

7.77778 

3.15348 

0.50918 

10 

0 

9 

9 

12 

6.33333 

3.08221 

0.56656 

10 

1 

9 

9 

13 

1.44444 

2.24227 

0.47140 

5 

0 

9 

9 

14 

6.11111 

2.52212 

0.62854 

10 

2 

9 

Weighted  Combined  Service  Class  Score  for  Class  9: 

Score: 6. 55620  Sigma :0 . 90845  Rho: 0.63213  evaluators : 9 .00000 


for 

Class 

10  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho  Max 

Min 

#Evaluators 

10 

1 

4.30000 

4.39823 

0.50000  10 

0 

10 

10 

2 

0.00000 

0.00000 

0.52915  0 

0 

10 

Weighted  Combined  Service  Class  Score  for  Class  10: 

Score: 2. 17562  Sigma: 2 . 52606  Rho: 1.24952  evaluators : 10.00000 
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CANDIDATE:  ARTX 


for  Class  11  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

11 

1 

0.08333 

0.28868 

0.47871 

1 

0 

12 

11 

2 

0.00000 

0.00000 

0.50000 

0 

0 

12 

11 

3 

0.50000 

1.44600 

0.47871 

5 

0 

12 

11 

4 

0.50000 

1.44600 

0.47871 

5 

0 

12 

11 

5 

0.25000 

0.62158 

0.47871 

2 

0 

12 

11 

6 

0.00000 

0.00000 

0.47871 

0 

0 

12 

11 

7 

0.00000 

0.00000 

0.47871 

0 

0 

12 

11 

8 

0.00000 

0.00000 

0.47871 

0 

0 

12 

11 

9 

0.33333 

0.77850 

0.47871 

2 

0 

12 

11 

10 

0.00000 

0.00000 

0.47871 

0 

0 

12 

11 

11 

0.41667 

1.44338 

0.47871 

5 

0 

12 

11 

12 

0.16667 

0.57735 

0.47871 

2 

0 

12 

11 

13 

0.83333 

1.94625 

0.52042 

5 

0 

12 

11 

14 

0.00000 

0.00000 

0.47871 

0 

0 

12 

Weighted  Combined  Service  Class  Score  for  Class  11: 

Score: 0.20744  Sigma: 0.22494  Rho: 0.13480  evaluators: 12.00000 


for  Class  12  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

12 

1 

0.00000 

0.00000 

0.51426 

0 

0 

11 

12 

2 

0.00000 

0.00000 

0.51426 

0 

0 

11 

12 

3 

7.18182 

3.12468 

0.61658 

10 

‘  0 

11 

12 

4 

0.00000 

0.00000 

0.51426 

0 

0 

11 

12 

5 

4.09091 

2.80908 

0.59613 

8 

0 

11 

12 

6 

5.81818 

2.99393 

0.63636 

10 

1 

11 

12 

7 

0.00000 

0.00000 

0.51426 

0 

0 

11 

12 

8 

0.00000 

0.00000 

0.51426 

0 

0 

11 

12 

9 

3.81818 

3.12468 

0.64922 

10 

0 

11 

12 

10 

4.00000 

3.46410 

0.64922 

10 

0 

11 

12 

11 

4.54545 

2.01810 

0.64922 

8 

2 

11 

12 

12 

1.27273 

2.05382 

0.57496 

5 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  12: 

Score:2. 62307  Sigma:0. 74252  Rho:0. 41133  evaluators: 11. 00000 
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NAVSWC  TR  90-246 


CANDIDATE:  ARTX 


for  Class  13  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

13 

1 

8.63636 

1.74773 

0.53009 

10 

5 

11 

13 

2 

8.45455 

1.96792 

0.50616 

10 

5 

11 

13 

3 

0.00000 

0.00000 

0.40656 

0 

0 

11 

13 

4 

7.45455 

2.42337 

0.50616 

10 

4 

11 

13 

5 

0.36364 

1.20605 

0.43598 

4 

0 

11 

13 

6 

4.90909 

1.86840  • 

0.57496 

10 

3 

11 

13 

7 

3.18182 

2.78633 

0.46355 

8 

0 

11 

13 

8 

4.27273 

1.27208 

0.50616 

5 

2 

11 

13 

9 

5.45455 

3.01210 

0.53009 

10 

0 

11 

13 

10 

6.36364 

2.61812 

0.50616 

10 

2 

11 

13 

11 

4.27273 

2.72363 

0.55298 

8 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  13: 

Score:5. 23006  Sigma:0. 85735  Rho:0. 55669  evaluators: 11. 00000 


for 

Class 

14  Mean  i 

scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

14 

1 

5.55556 

3.35824 

0.68493 

10 

0 

9 

14 

2 

5.11111 

3.40751 

0.72860 

10 

0 

9 

14 

3 

2.44444 

2.87711 

0.80890 

8 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  14: 

Score: 4. 51381  Sigma: 2. 00623  Rho: 0.77566  evaluators : 9 . 00000 


for  Class  15  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

15 

1 

5.61538 

1.55662 

0.48038 

10 

4 

13 

15 

2 

6.69231 

3.03822 

0.46154 

10 

0 

13 

15 

3 

0.38462 

1.38675 

0.38462 

5 

0 

13 

15 

4 

0.00000 

0.00000 

0.38462 

0 

0 

13 

15 

5 

0.00000 

0.00000 

0.38462 

0 

0 

13 

15 

6 

2.23077 

2.52170 

0.48038 

5 

0 

13 

15 

7 

3.53846 

2.98930 

0.51602 

10 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  15: 

Score:2. 96454  Sigma:0. 91737  Rho:0. 43637  evaluators: 13. 00000 
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CANDIDATE:  ARTX 


for 

Class 

16  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

16 

1 

8.55556 

2.24227 

0.54433 

10 

5 

9 

16 

2 

9.00000 

2.00000 

0.50918 

10 

5 

9 

16 

3 

9.44444 

1.66667 

0.53287 

10 

5 

9 

16 

4 

9.44444 

1.66667 

0.47140 

10 

5 

9 

16 

5 

8.66667 

2.17945 

0.56656 

10 

5 

9 

16 

6 

1.11111 

2.20479 

0.53287 

5 

0 

9 

16 

7 

9.11111 

1.83333 

0.50918 

10 

5 

9 

16 

8 

9.44444 

1.66667 

0.50918 

10 

5 

9 

16 

9 

8.22222 

2.43812 

0.59835 

10 

5 

9 

16 

10 

5.44444 

3.53946 

0.64788 

10 

0 

9 

16 

11 

4.66667 

3.00000 

0.59835 

8 

0 

9 

16 

12 

5.00000 

3.42783 

0.56656 

10 

0 

9 

16 

13 

9.22222 

1.71594 

0.61864 

10 

5 

9 

16 

14 

6.00000 

3.16228 

0.67586 

10 

0 

9 

16 

15 

6.44444 

2.74368 

0.64788 

10 

3 

9 

16 

16 

5.44444 

3.87657 

0.59835 

10 

0 

9 

16 

17 

3.44444 

4.27525 

0.59835 

10 

0 

9 

16 

18 

3.44444 

4.27525 

0.59835 

10 

0 

9 

16 

19 

3.88889 

3.65529 

0.64788 

10 

0 

9 

16 

20 

5.77778 

3.38296 

0.67586 

10 

2 

9 

Weighted  Combined  Service  Class  Score  for  Class  16: 

Score: 6. 84386  Sigma: 0.87917  Rho: 0.63015  evaluators : 9 .00000 
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CANDIDATE:  CRONUS 


Results  with  data  for  evaluations  with  no  less  than  7  candidates  evaluated. 
221  files  for  candidate  CRONUS 

for  Class  0  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Kin 

#Evaluators 

0 

1 

7.40000 

1.98415 

0.40620 

10 

4 

20 

0 

2 

5.90000 

1.97084 

0.45552 

10 

2 

20 

0 

3 

7.00000 

2.00000 

0.43012 

10 

4 

20 

0 

4 

7.05000 

1.95946 

0.44159 

10 

3 

20 

0 

5 

4.80000 

2.60768 

0.46098 

10 

0 

20 

0 

6 

7.75000 

1.97017 

0.41533 

10 

4 

20 

0 

7 

5.45000 

2.45967 

0.46098 

9 

0 

20 

0 

8 

6.00000 

2.57519 

0.49749 

10 

0 

20 

Mean  Service  Class  Score  for  Class  0: 


Score:6. 41875  Sigma:0. 78086  Rho:0. 15799  evaluators: 20. 00000 
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NAVSWC  TR  90-246 


CANDIDATE:  CRONUS 

for  Class  1  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

1  1  5.42222  1.97126  0.26759  10  2  45 

1  2  7.44444  2.01760  0.28109  10  2  45 

1  3  6.13333  1.86596  0.28717  10  2  45 

1  4  8.64444  1.90878  0.26851  10  3  45 

1  5  7.64444  2.21724  0.30872  10  1  45 

1  6  8.48889  1.56121  0.28889  10  5  45 

1  7  8.02222  2.08336  0.29144  10  1  45 

1  8  5.55556  2.98142  0.30144  10  0  45 

1  9  6.88889  2.69024  0.30631  10  0  45 

1  10  2.82222  2.69080  0.31348  10  0  45 

1  11  7.68889  3.03631  0.26105  10  0  45 

1  12  7.02222  2.16888  0.31427  10  2  45 

1  13  7.57778  2.64995  0.28889  10  0  45 

1  14  7.37778  2.41481  0.33993  10  2  45 

1  15  6.80000  2.13839  0.30631  10  0  45 

1  16  8.11111  2.00252  0.31427  10  2  45 

1  17  7.24444  2.46019  0.29565  10  0  45 

1  18  8.48889  1.99570  0.29979  10  0  45 

1  19  4.86667  3.08663  0.33921  10  0  45 

1  20  6.04444  3.25406  0.28458  10  0  45 

1  21  1.24444  2.01309  0.30225  7  0  45 

1  22  6.71111  3.04229  0.29979  10  0  45 

1  23  2.84444  2.61078  0.29481  10  0  45 

1  24  7.60000  2.45320  0.28974  10  0  45 

1  25  9.02222  1.95969  0.26387  10  0  45 

1  26  8.08889  2.63561  0.28889  10  0  45 

1  27  7.77778  2.33442  0.31427  10  0  45 

1  28  1.84444  2.94615  0.28109  10  0  45 

1  29  1.73333  2.98785  0.28545  10  0  45 

Mean  Service  Class  Score  for  Class  1: 

Score: 6. 38739  Sigma: 0.68018  Rho: 0.50590  evaluators: 45. 00000 

for  Class  2  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

2  1  5.11111  3.72305  0.65734  10  0  9 

Weighted  Combined  Service  Class  Score  for  Class  2: 

Score: 5. 11111  Sigma: 3.72305  Rho: 0.65734  evaluators: 9. 00000 
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CANDIDATE:  CRONUS 


for  Class  3  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

3 

1 

7.71429 

2.98408 

0.51508 

10 

2 

7 

3 

2 

7.57143 

2.76026 

0.51508 

10 

2 

7 

3 

3 

8.28571 

2.05866 

0.51508 

10 

5 

7 

3 

4 

7.85714 

2.67261 

0.51508 

10 

3 

7 

3 

5 

6.14286 

3.93398 

0.65465 

10 

0 

7 

3 

6 

8.57143 

1.81265 

0.57143 

10 

5 

7 

3 

7 

8.57143 

1.90238 

0.57143 

10 

5 

7 

3 

8 

7.85714 

3.57904 

0.57143 

10 

0 

7 

3 

9 

6.28571 

3.49830 

0.65465 

10 

0 

7 

3 

10 

6.28571 

3.49830 

0.65465 

10 

0 

7 

3 

11 

7.00000 

3.36650 

0.57143 

10 

0 

7 

3 

12 

4.14286 

4.33699 

0.45175 

10 

0 

7 

3 

13 

7.42857 

2.14920 

0.51508 

10 

4 

7 

3 

14 

7.14286 

2.03540 

0.57143 

10 

4 

7 

3 

15 

5.71429 

2.98408 

0.51508 

10 

2 

7 

3 

16 

6.00000 

2.76887 

0.51508 

10 

2 

7 

3 

17 

4.28571 

4.27061 

0.45175 

10 

0 

7 

3 

18 

6.85714 

3.93398 

0.57143 

10 

0 

7 

3 

19 

5.28571 

3.35233 

0.51508 

10 

1 

7 

3 

20 

3.71429 

4.68025 

0.51508 

10 

0 

7 

3 

21 

7.57143 

2.63674 

0.57143 

10 

3 

7 

3 

22 

4.14286 

4.41318 

0.45175 

10 

0 

7 

3 

23 

4.42857 

4.39155 

0.51508 

10 

0 

7 

3 

24 

3.71429 

4.68025 

0.45175 

10 

0 

7 

Weighted  Combined  Service  Class  Score  for  Class  3: 

Score: 6. 48214  Sigma: 0.72539  Rho: 0.11907  evaluators: 7.00000 


for  Class  4  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

4  1  8.33333  2.50000  0.67586  10  5  9 

Weighted  Combined  Service  Class  Score  for  Class  4: 

Score: 8. 33333  Sigma: 2. 50000  Rho: 0.67586  evaluators: 9. 00000 


for  Class  5  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

5 

1 

3.23077 

2.38586 

5 

2 

5.92308 

3.14806 

5 

3 

5.00000 

3.13581 

5 

4 

6.15385 

3.78255 

5 

5 

0.53846 

1.05003 

5 

6 

1.76923 

2.16617 

Weighted  Combined  Service  Class 
Score: 3. 71087  Sigma: 1.26076  Rho 


Rho 

Max  Min 

#Evaluators 

0.61056 

7 

0 

13 

0.58583 

10 

X 

13 

0.53294 

10 

0 

13 

0.54393 

10 

0 

13 

0.57048 

3 

0 

13 

0.61056 

5 

0 

13 

Score  for 

Class 

5: 

0.64973  evaluators: 13. 00000 
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CANDIDATE:  CRONUS 


for  Class  6  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

6 

1 

3.62500 

4.24054 

0.82916 

10 

0 

8 

6 

2 

1.12500 

1.80772 

0.75000 

5 

0 

8 

6 

3 

2.12500 

3.64251 

0.69597 

10 

0 

8 

6 

4 

4.62500 

5.04090 

0.78062 

10 

0 

8 

6 

5 

2.37500 

3.37797 

0.76035 

8 

0 

8 

6 

6 

2.37500 

3.37797 

0.76035 

8 

0 

8 

6 

7 

4.87500 

5.22186 

0.69597 

10 

0 

8 

6 

8 

5.00000 

5.34522 

0.66144 

10 

0 

8 

6 

9 

4.75000 

5.11999 

0.69597 

10 

0 

8 

6 

10 

5.00000 

5.34522 

0.69597 

10 

0 

8 

6 

11 

5.00000 

5.34522 

0.69597 

10 

0 

8 

6 

12 

3.62500 

4.40576 

0.78062 

10 

0 

8 

6 

13 

3.62500 

4.40576 

0.76035 

10 

0 

8 

6 

14 

3.87500 

4.48609 

0.76035 

10 

0 

8 

6 

15 

1.25000 

2.31455 

0.69597 

5 

0 

8 

6 

16 

5.00000 

5.34522 

0.69597 

10 

0 

8 

6 

17 

3.50000 

4.40779 

0.78062 

10 

0 

8 

6 

18 

2.25000 

2.49285 

0.81009 

5 

0 

8 

6 

19 

4.50000 

4.98569 

0.69597 

10 

0 

8 

6 

20 

2.87500 

4.18970 

0.78062 

10 

0 

8 

Weighted  Combined  Service  Class  Score  for  Class  6: 

Score: 3. 72251  Sigma: 1.06547  Rho: 0.30213  evaluators: 8. 00000 


for  Class  7  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

7 

1 

0.00000 

0,00000 

0.66144 

0 

0 

8 

7 

2 

0.62500 

1.76777 

0.59948 

5 

0 

8 

7 

3 

0.62500 

1.76777 

0.59948 

5 

0 

8 

7 

4 

0.62500 

1.76777 

0.59948 

5 

0 

8 

7 

5 

0.62500 

1.76777 

0.59948 

5 

0 

S 

7 

6 

0.50000 

1.41421 

0.59948 

4 

0 

8 

7 

7 

0.62500 

1.76777 

0.59948 

5 

0 

8 

7 

8 

1.00000 

1.92725 

0.59948 

5 

0 

8 

7 

9 

0.00000 

0.00000 

0.59948 

0 

0 

8 

7 

10 

0.62500 

1.76777 

0.59948 

5 

0 

8 

7 

11 

0.62500 

1.76777 

0.59948 

5 

0 

8 

Weighted  Combined  Service  Class  Score  for  Class  7: 

Score:0. 52556  Sigma:0. 48067  Rho:0. 19286  evaluators: 8. 00000 
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CANDIDATE:  CRONUS 


for  Class  8  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

8 

1 

2.36364 

3.07482 

0.48956 

10 

0 

11 

8 

2 

3.18182 

3.89405 

0.48956 

10 

0 

11 

8 

3 

8.90909 

1.70027 

0.53009 

10 

5 

11 

8 

4 

7.27273 

3.95198 

0.62324 

10 

0 

11 

8 

5 

5.27273 

3.79713 

0.60984 

10 

0 

11 

8 

6 

7.90909 

3.17662 

0.58916 

10 

0 

11 

8 

7 

8.81818 

1.83402 

0.55298 

10 

5 

11 

8 

8 

7.90909 

3.14498 

0.60984 

10 

0 

11 

8 

9 

5.63636 

3.47197 

0.62984 

10 

0 

11 

8 

10 

3.18182 

3.84235 

0.66183 

10 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  8: 

Score: 6. 03731  Sigma: 1.38629  Rho: 0.94291  evaluators: 11.00000 


for  Class  9  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

9 

1 

6.11111 

1.76383 

0.62854 

8 

3 

9 

9 

2 

6.77778 

2.33333 

0.62854 

10 

3 

9 

9 

3 

2.44444 

3.53946 

0.54433 

10 

0 

9 

9 

4 

1.00000 

1.73205 

0.54433 

5 

0 

9 

9 

5 

2.44444 

3.53946 

0.54433 

10 

0 

9 

9 

6 

2.44444 

3.53946 

0.54433 

10 

0 

9 

9 

7 

0.77778 

1.71594 

0.54433 

5 

0 

9 

9 

8 

8.33333 

2.12132 

0.57735 

10 

5 

9 

9 

9 

5.00000 

2.73861 

0.62854 

10 

1 

9 

9 

10 

4.88889 

2.71314 

0.62854 

10 

1 

9 

9 

11 

0.66667 

1.41421 

0.50918 

4 

0 

9 

9 

12 

8.88889 

1.69148 

0.54433 

10 

5 

9 

9 

13 

2.11111 

3.21887 

0.54433 

9 

0 

9 

9 

14 

0.77778 

1.71594 

0.50918 

5 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  9: 

Score: 3. 92302  Sigma: 0.80059  Rho: 0.44303  evaluators: 9. 00000 


for  Class 

10  Mean  i 

scores  for 

each  Criterion  are: 

Cl  Cr 

Score 

Std  dev 

Rho  Max  Min 

#Evaluators 

10  1 

1.40000 

1.83787 

0.48990  5  0 

10 

10  2 

1.40000 

2.06559 

0.48990  5  0 

10 

Weighted  Combined  Service  Class  Score  for  Class  10: 

Score: 1.40000  Sigma: 1.47631  Rho: 0.62722  evaluators: 10. 00000 
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CANDIDATE:  CRONUS 


for  Class  11  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

11 

1 

4.83333 

4.30292 

0.51370 

10 

0 

12 

11 

2 

1.66667 

3.67630 

0.47871 

10 

0 

12 

11 

3 

1.08333 

2.60971 

0.47871 

8 

0 

12 

11 

4 

1.16667 

2.85509 

0.47871 

9 

0 

12 

11 

5  . 

3.41667 

2.46644 

0.57130 

8 

0 

12 

11 

6 

1.75000 

3.46738 

0.50000 

10 

0 

12 

11 

7 

1.08333 

2.87492 

0.51370 

10 

0 

12 

11 

8 

1.58333 

3.17543 

0.53359 

10 

0 

12 

11 

9 

4.16667 

3.21455 

0.61237 

10 

0 

12 

11 

10 

0.50000 

1.44600 

0.50000 

5 

0 

12 

11 

11 

0.58333 

1.08362 

0.54645 

3 

0 

12 

11 

12 

0.25000 

0.86603 

0.51370 

3 

0 

12 

11 

13 

0.25000 

0.86603 

0.54645 

3 

0 

12 

11 

14 

1.25000 

3.10791 

0.47871 

10 

0 

12 

Veighted  Combined  Service  Class  Score  for  Class  11: 

Score:l. 80940  Sigma:0. 81801  Rho:0. 27765  evaluators : 12 . 00000 


for  Class 

12  Mean  i 

scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

12 

1 

0.18182 

0.60302 

0.51426 

2 

0 

11 

12 

2 

0.18182 

0.60302 

0.51426 

2 

0 

11 

12 

3 

3.27273 

3.77070 

0.61658 

10 

0 

11 

12 

4 

0.18182 

0.60302 

0.51426 

2 

0 

11 

12 

5 

0.63636 

1.56670 

0.51426 

5 

0 

11 

12 

6 

0.18182 

0.60302 

0.55298 

2 

0 

11 

12 

7 

0.18182 

0.60302 

0.53783 

2 

0 

11 

12 

8 

0.18182 

0.60302 

0.53783 

2 

0 

11 

12 

9 

•  0.27273 

0.64667 

0.53783 

2 

0 

11 

12 

10 

0.27273 

0.64667 

0.53783 

2 

0 

11 

12 

11 

0.18182 

0.60302 

0.51426 

2 

0 

11 

12 

12 

0.63636 

1.56670 

0.51426 

5 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  12: 

Score:0. 57220  Sigma:0. 44313  Rho:0. 17283  evaluators: 11. 00000 
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CANDIDATE:  CRONUS 


.for  Class  13  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

13 

1 

6.63636 

2.69343 

0.58916 

10 

3 

11 

13 

2 

6.81818 

2.85721 

0.58916 

10 

3 

11 

13 

3 

0.90909 

3.01511 

0.45455 

10 

0 

11 

13 

4 

4.45455 

4.29799 

0.62984 

10 

0 

11 

13 

5 

1.90909 

3.20794 

0.56773 

10 

0 

11 

13 

6 

3.09091 

3.67300 

0.57496 

10 

0 

11 

13 

7 

1.81818 

2.99393 

0.60984 

8 

0 

11 

13 

8 

2.27273 

3.00303 

0.62984 

8 

0 

11 

13 

9 

2.27273 

3.00303 

0.60984 

8 

0 

11 

13 

10 

4.27273 

3.37908 

0.60984 

10 

0 

11 

13 

11 

0.00000 

0.00000 

0.50616 

0 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  13: 

Score: 3. 38001  Sigma: 1.00068  Rho: 0.39158  evaluators: 11.00000 


for 

Class 

14  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

14 

1 

2.00000 

3.50000 

0.54433 

10 

0 

9 

14 

2 

4.66667 

4.55522 

0.65734 

10 

0 

9 

14 

3 

4.88889 

4.72875 

0.60858 

10 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  14: 

Score: 3. 78062  Sigma: 2 . 55193  Rho: 0.76695  evaluators : 9 . 00000 


for 

Class 

15  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

15 

1 

4.92308 

3.25222 

0.53846 

10 

0 

13 

15 

2 

0.15385 

0.55470 

0.47419 

2 

0 

13 

15 

3 

2.15385 

2.51151 

0.55470 

6 

0 

13 

15 

4 

2.92308 

3.98877 

0.52736 

10 

0 

13 

15 

5 

0.00000 

0.00000 

0.49255 

0 

0 

13 

15 

6 

2.23077 

2.97640 

0.52736 

10 

0 

13 

15 

7 

2.07692 

2.21591 

0.46154 

5 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  15: 

Score: 2. 17131  Sigma: 1.04210  Rho: 0.35983  evaluators: 13. 00000 
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CANDIDATE:  CRONUS 


for 

Class 

16  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

16 

1 

2.22222 

3.63242 

0.62854 

10 

0 

9 

16 

2 

2.00000 

3.50000 

0.67586 

10 

0 

9 

16 

3 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

4 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

5 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

6 

2.22222 

3.63242 

0.62854 

10 

0 

9 

16 

7 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

8 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

9 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

10 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

11 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

12 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

13 

2.00000 

3.50000 

0.67586 

10 

0 

9 

16 

14 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

15 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

16 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

17 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

18 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

19 

2.00000 

3.50000 

0.62854 

10 

0 

9 

16 

20 

2.00000 

3.50000 

0.62854 

10 

0 

9 

eighted  Combined  Service  Class  Score  for 

Class  16: 

Score: 2. 01992  Sigma: 0.81556  Rho: 0.23066  evaluators: 9. 00000 
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CANDIDATE:  IRMX 


Results  with  data  for  evaluations  with  no  less  than  7  candidates  evaluated. 
216  files  for  candidate  IRMX 

for  Class  0  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

0 

1 

5.45000 

1.19097 

0.39686 

10 

5 

20 

0 

2 

5.55000 

2.43818 

0.49497 

10 

1 

20 

0 

3 

7.80000 

1.76516 

0.43012 

10 

5 

20 

0 

4 

8.70000 

1.55935 

0.38730 

10 

5 

20 

0 

5 

6.30000 

2.61775 

0.47434 

10 

0 

20 

0 

6 

8.15000 

1.92696 

0.47958 

10 

5 

20 

0 

7 

7.40000 

2.03651 

0.46904 

10 

5 

20 

0 

8 

5.60000 

2.43656 

0.53385 

9 

0 

20 

Mean  Service  Class  Score  for  Class  0: 

Score:6. 86875  Sigma: 0.72414  Rho:0. 16286  evaluators: 20. 00000 


E-25 


NAVSWC  TR  90-246 


CANDIDATE:  IRMX 


for  Class  1  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

1 

1 

7.33333 

1.59545 

0.27666 

10 

5 

45 

1 

2 

7.33333 

1.93061 

0.27756 

10 

3 

45 

1 

3 

7.15556 

1.74454 

0.29397 

10 

5 

45 

1 

4 

5.77778 

2.24508 

0.29648 

10 

0 

45 

1 

5 

8.17778 

1.70945 

0.29313 

10 

4 

45 

1 

6 

8.42222 

1.81520 

0.28109 

10 

5 

45 

1 

7 

8.17778 

1.74888 

0.28631 

10 

5 

45 

1 

8 

8.22222 

1.70412 

0.28458 

10 

5 

45 

1 

9 

6.46667 

2.32183 

0.29481 

10 

0 

45 

1 

10 

4.66667 

2.72196 

0.28197 

10 

0 

45 

1 

11 

7.88889 

2.49747 

0.27126 

10 

1 

45 

1 

12 

8.42222 

1.68535 

0.30388 

10 

5 

45 

1 

13 

7.86667 

1.97254 

0.29313 

10 

1 

45 

1 

14 

8.13333 

1.92590 

0.33036 

10 

4 

45 

1 

15 

7.88889 

1.81186 

0.31427 

10 

5 

45 

1 

16 

8.40000 

1.64317 

0.32356 

10 

4 

45 

1 

17 

8.00000 

1.90693 

0.29481 

10 

3 

45 

1 

18 

8.62222 

1.69610 

0.29229 

10 

5 

45 

1 

19 

7.40000 

2.78307 

0.31817 

10 

0 

45 

1 

20 

8.53333 

1.87810 

0.27126 

10 

5 

45 

1 

21 

5.26667 

2.80746 

0.31427 

10 

0 

45 

1 

22 

7.75556 

2.34671 

0.29979 

10 

0 

45 

1 

23 

5.82222 

2.55208 

0.29979 

10 

0 

45 

1 

24 

7.04444 

1.90640 

0.28803 

10 

3 

45 

1 

25 

7.17778 

1.92223 

0.29979 

10 

4 

45 

1 

26 

7.84444 

2.04446 

0.28197 

10 

2 

45 

1 

27 

3.66667 

3.11156 

0.29313 

10 

0 

45 

1 

28 

6.53333 

2.49180 

0.29979 

10 

0 

45 

1 

29 

6.02222 

2.76742 

0.30551 

10 

0 

45 

Mean  Service  Class  Score  for  Class  1: 

Score: 7. 18956  Sigma: 0,68293  Rho: 0.55192  evaluators: 45. 00000 


for  Class  2  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

2  1  4.66667  1.50000  0.60858  729 

Weighted  Combined  Service  Class  Score  for  Class  2: 

Score: 4. 66667  Sigma: 1.50000  Rho: 0.60858  evaluators: 9. 00000 
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NAVSWC  TR  90-246 


CANDIDATE:  IRMX 


for  Class  3  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

3 

1 

1.42857 

1.98806 

0.45175 

5 

0 

7 

3 

2 

1.00000 

1.91485 

0.45175 

5 

0 

7 

3 

3 

0.42857 

1.13389 

0.45175 

3 

0 

7 

3 

4 

0.42857 

1.13389 

0.45175 

3 

0 

7 

3 

5 

0.71429 

1.25357 

0.45175 

3 

0 

7 

3 

6 

2.14286 

3.18479 

0.51508 

8 

0 

7 

3 

7 

1.14286 

1.57359 

0.45175 

4 

0 

7 

3 

8 

0.71429 

1.25357 

0.45175 

3 

0 

7 

3 

9 

0.57143 

0.97590 

0.45175 

2 

0 

7 

3 

10 

0.85714 

1.06904 

0.45175 

2 

0 

7 

3 

11 

1.28571 

1.88982 

0.45175 

5 

0 

7 

3 

12 

0.85714 

1.06904 

0.45175 

2 

0 

7 

3 

13 

1.28571 

1.70434 

0.45175 

4 

0 

7 

3 

14 

1.00000 

1.29099 

0.45175 

3 

0 

7 

3 

15 

1.14286 

1.46385 

0.45175 

3 

0 

7 

3 

16 

1.42857 

1.98806 

0.45175 

5 

0 

7 

3 

17 

0.42857 

1.13389 

0.45175 

3 

0 

7 

3 

18 

1.14286 

2.03540 

0.45175 

5 

0 

7 

3 

19 

1.14286 

2.03540 

0.45175 

5 

0 

7 

3 

20 

1.42857 

3.77964 

0.45175 

10 

0 

7 

3 

21 

0.71429 

1.88982 

0.45175 

5 

0 

7 

3 

22 

0.57143 

1.51186 

0.55328 

4 

0 

7 

3 

23 

0.28571 

0.75593 

0.45175 

2 

0 

7 

3 

24 

0.71429 

1.88982 

0.45175 

5 

0 

7 

Weighted  Combined  Service  Class  Score  for  Class  3: 

Score: 0.96429  Sigma: 0.38409  Rho: 0.10363  evaluators: 7. 00000 


for  Class  4  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

4  1  1.00000  3.00000  0.50918  909 

Weighted  Combined  Service  Class  Score  for  Class  4: 

Score: 1.00000  Sigma: 3.00000  Rho: 0.50918  evaluators: 9. 00000 


for  Class  5  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

5 

1 

6.53846 

2.25889 

5 

2 

7.15385 

2.30384 

5 

3 

6.23077 

2.74329 

5 

4 

1.61538 

1.85016 

5 

5 

2.92308 

3.22649 

5 

6 

7.23077 

3.70031 

Weighted  Combined  Service  Class 
Score: 5. 42204  Sigma: 1.41020  Rho 


Rho 

Max  Min 

#Evaluators 

0.44853 

10 

4 

13 

0.48038 

10 

4 

13 

0.49852 

10 

0 

13 

0.48038 

5 

0 

13 

0.54393 

8 

0 

13 

0.48038 

10 

0 

13 

Score  for 

Class 

5: 

0.83903  evaluators: 13. 000 
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NAVSWC  TR  90-246 


CANDIDATE:  IRHX 


for  Class  6  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

6 

1 

9.12500 

1.80772 

0.59948 

10 

5 

8 

6 

2 

8.62500 

2.66927 

0.66144 

10 

3 

8 

6 

3 

4.12500 

4.08613 

0.59948 

10 

0 

8 

6 

4 

8.25000 

2.05287 

0.75000 

10 

5 

8 

6 

5 

6.00000 

2.92770 

0.69597 

10 

0 

8 

6 

6 

2.00000 

1.92725 

0.71807 

6 

0 

8 

6 

7 

9.12500 

1.24642 

0.63738 

10 

7 

8 

6 

8 

9.50000 

0.92582 

0.63738 

10 

8 

8 

6 

9 

9.12500 

1.45774 

0.59948 

10 

6 

8 

6 

10 

9.75000 

0.46291 

0.59948 

10 

9 

f* 

v> 

6 

11 

7.75000 

3.57571 

0.55902 

10 

0 

8 

6 

12 

9.87500 

0.35355 

0.55902 

10 

9 

8 

6 

13 

9.87500 

0.35355 

0.55902 

10 

9 

8 

6 

14 

9.87500 

0.35355 

0.51539 

10 

9 

8 

6 

15 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

16 

8.37500 

3.46152 

0.55902 

10 

0 

8 

6 

17 

9.50000 

0.92582 

0.55902 

10 

8 

8 

6 

18 

9.00000 

1.19523 

0.62500 

10 

7 

8 

6 

19 

8.37500 

3.54310 

0.55902 

10 

0 

8 

6 

20 

7.62500 

3.33542 

0.51539 

10 

1 

8 

Weighted  Combined  Service  Class  Score  for  Class  6: 

Score: 8. 09092  Sigma: 0.76040  Rho: 0.58874  evaluators : 8 . 00000 


for  Class  7  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max  Min 

#Evaluators 

7 

1 

6.37500 

3.06769 

0.72887 

10 

2 

8 

7 

2 

8.75000 

2.05287 

0.72887 

10 

5 

8 

7 

3 

9.37500 

1.76777 

0.63738 

10 

5 

8 

7 

4 

9.00000 

1.77281 

0.67315 

10 

5 

8 

7 

5 

9.12500 

1.80772 

0.67315 

10 

5 

8 

7 

6 

7.87500 

2.99702 

0.81009 

10 

3 

8 

7 

7 

8.25000 

2.05287 

0.72887 

10 

4 

8 

7 

8 

9.12500 

1.24642 

0.69597 

10 

7 

8 

7 

9 

9.25000 

1.16496 

0.66144 

10 

7 

8 

7 

10 

9.75000 

0.70711 

0.76035 

10 

8 

8 

7 

11 

5.12500 

3.64251 

0.63738 

10 

0 

8 

Weighted  Combined  Service  Class 

Score  for 

Class 

7: 

Score:8. 40252  Sigma:0. 98897  Rho:0. 76363  evaluators: 8. 00000 
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NAVSWC  TR  90-246 


CANDIDATE:  IRMX 


for  Class  8  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

8 

1 

3.18182 

2.22792 

0.46355 

5 

0 

11 

8 

2 

7.90909 

2.62505 

0.48956 

10 

3 

11 

8 

3 

9.00000 

1.41421 

0.48956 

10 

7 

11 

8 

4 

1.81818 

3.62817 

0.46355 

10 

0 

11 

8 

5 

3.27273 

4.60632 

0.55298 

10 

0 

11 

8 

6 

7.36364 

3.20227 

0.51426 

10 

0 

11 

8 

7 

8.27273 

2.96954 

0.46355 

10 

2 

11 

8 

8 

6.27273 

2.90141 

0.56040 

10 

2 

11 

8 

9 

0.00000 

0.00000 

0.43598 

0 

0 

11 

8 

10 

1.36364 

3.23335 

0.58210 

10 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  8: 

Score:5. 05986  Sigma:l. 21205  Rho:0. 80796  evaluators: 11. 00000 


for  Class  9  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

9 

1 

9.22222 

1.09291 

0.47140 

10 

7 

9 

9 

2 

9.55556 

1.01379 

0.47140 

10 

7 

9 

9 

3 

7.22222 

2.10819 

0.47140 

10 

5 

9 

9 

4 

6.00000 

3.12250 

0.61864 

10 

0 

9 

9 

5 

9.22222 

1.30171 

0.56656 

10 

7 

9 

9 

6 

9.22222 

1.30171 

0.56656 

10 

7 

9 

9 

7 

6.44444 

1.58990 

0.59835 

9 

5 

9 

9 

8 

9.33333 

1.00000 

0.47140 

10 

7 

9 

9 

9 

9.22222 

0.83333 

0.47140 

10 

8 

9 

9 

10 

7.55556 

1.58990 

0.47140 

10 

5 

9 

9 

11 

8.44444 

2.06828 

0.47140 

10 

5 

9 

9 

12 

8.55556 

3.28295 

0.43033 

10 

0 

9 

9 

13 

1.44444 

3.35824 

0.47140 

10 

0 

9 

9 

14 

9.55556 

0.72648 

0.50918 

10 

8 

9 

Weighted  Combined  Service  Class  Score  for  Class  9: 

Score:8. 08862  Sigma:0. 91287  Rho:0. 77591  evaluators: 9. 00000 


for  Class 

10  Mean  ; 

scores  for 

each  Criterion 

are : 

Cl  Cr 

Score 

Std  dev 

Rho  Max 

Min 

#Evaluators 

10  1 

8.60000 

1.77639 

0.61644  10 

5 

10 

10  2 

0.40000 

0.84327 

0.50000  2 

0 

10 

Weighted  Combined  Service  Class  Score  for  Class  10: 

Score:4. 54885  Sigma: 2 . 58982  Rho:2. 42572  evaluators: 10. 00000 
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NAVSWC  TR  90-246 


CANDIDATE:  IRMX 


for 

Class 

11  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

11 

1 

6.41667 

2.71221 

0.47871 

10 

2 

12 

11 

2 

6.83333 

2.82307 

0.47871 

10 

2 

12 

11 

3 

3.25000 

1.54479 

0.52042 

5 

1 

12 

11 

4 

3.33333 

1.77525 

0.55277 

5 

1 

12 

11 

5 

3.58333 

1.67649 

0.55277 

6 

1 

12 

11 

6 

3.75000 

2.92715 

0.52042 

8 

0 

12 

11 

7 

5.25000 

3.33371 

0.51370 

10 

0 

12 

11 

8 

3.58333 

1.92865 

0.52042 

6 

0 

12 

11 

9 

5.41667 

3.05877 

0.55277 

10 

0 

12 

11 

10 

4.00000 

3.46410 

0.47871 

10 

0 

12 

11 

11 

0.41667 

1.16450 

0.47871 

4 

0 

12 

11 

12 

4.83333 

3.04014 

0.51370 

10 

1 

12 

11 

13 

6.33333 

2.93361 

0.47871 

10 

2 

12 

11 

14 

0.33333 

1.15470 

0.40825 

4 

0 

12 

Weighted  Combined  Service  Class  Score  for  Class  11: 

Score: 4. 15814  Sigma: 0.84425  Rho: 0.51185  evaluators: 12. 00000 


for  Class  12  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

12 

1 

0.45455 

0.82020 

0.57496 

2 

0 

11 

12 

2 

1.81818 

2.35874 

0.53783 

7 

0 

11 

12 

3 

8.18182 

1.88776 

0.53783 

10 

5 

11 

12 

4 

6.90909 

1.64040 

0.56040 

10 

5 

11 

12 

5 

4.54545 

3.38714 

0.62984 

10 

0 

11 

12 

6 

6.54545 

2.76997 

0.59613 

10 

3 

11 

12 

7 

0.00000 

0.00000 

0.57496 

0 

0 

11 

12 

8 

0.00000 

0.00000 

0.57496 

0 

0 

11 

12 

9 

0.54545 

1.80907 

0.57496 

6 

0 

11 

12 

10 

0.54545 

1.80907 

0.57496 

6 

0 

11 

12 

11 

5.63636 

2.33550 

0.61658 

9 

2 

11 

12 

12 

3.00000 

2.32379 

0.61658 

6 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  12: 

Score:3. 25912  Sigma:0. 73056  Rho:0. 45889  evaluators: 11. 00000 


NAVSWC  TR  90-246 


CANDIDATE:  IRMX 


for  Class  13  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

13 

1 

8.90909 

1.44600 

0.46355 

10 

6 

11 

13 

2 

8.63636 

1.96330 

0.50616 

10 

5 

11 

13 

3 

3.63636 

2.73030 

0.53009 

8 

0 

11 

13 

4 

8.63636 

1.74773 

0.48956 

10 

5 

11 

13 

5 

2.09091 

2.25630 

0.46355 

5 

0 

11 

13 

6 

6.27273 

3.55221 

0.48956 

10 

0 

11 

13 

7 

5.72727 

2.45320 

0.53009 

10 

2 

11 

13 

8 

6.54545 

2.62159 

0.51426 

10 

2 

11 

13 

9 

5.45455 

2.94495 

0.48956 

10 

1 

11 

13 

10 

6.36364 

3.13920 

0.46355 

10 

0 

11 

13 

11 

4.09091 

2.11918 

0.48956 

7 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  13: 

Score: 6. 31206  Sigma: 0.96023  Rho: 0.63614  evaluators: 11. 00000 


for 

Class 

14  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

14 

1 

8.33333 

2.17945 

0.68493 

10 

5 

9 

14 

2 

7.44444 

3.84419 

0.74536 

10 

0 

9 

14 

3 

3.66667 

3.08221 

0.88192 

10 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  14: 

Score: 6. 69054  Sigma: 2. 05181  Rho: 1.05644  evaluators : 9 . 00000 


for 

Class 

15  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

15 

1 

5.61538 

1.19293 

0.40704 

8 

5 

13 

15 

2 

6.07692 

3.22649 

0.42829 

10 

0 

13 

15 

3 

3.07692 

2.32600 

0.46154 

5 

0 

13 

15 

4 

3.15385 

3.82636 

0.48038 

10 

0 

13 

15 

5 

3.61538 

3.73136 

0.49255 

10 

0 

13 

15 

6 

6.92308 

2.95696 

0.44853 

10 

0 

13 

15 

7 

4.23077 

1.83275 

0.38462 

7 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  15: 

Score:4. 87557  Sigma:l. 18810  Rho:0. 59965  evaluators: 13.00000 
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CANDIDATE:  IRMX 


for 

Class 

16  Mean  : 

scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

16 

1 

7.22222 

2.33333 

0.60858 

10 

5 

9 

16 

2 

6.55556 

3.32081 

0.60858 

10 

0 

9 

16 

3 

7.77778 

2.63523 

0.60858 

10 

5 

9 

16 

4 

7.66667 

2.78388 

0.65734 

10 

4 

9 

16 

5 

7.44444 

2.65100 

0.65734 

10 

4 

9 

16 

6 

5.44444 

3.28295 

0.65734 

10 

0 

9 

16 

7 

4.88889 

2.84800 

0.68493 

10 

0 

9 

16 

8 

4.88889 

2.84800 

0.68493 

10 

0 

9 

16 

9 

7.44444 

2.40370 

0.57735 

10 

5 

9 

16 

10 

4.88889 

1.83333 

0.70273 

8 

2 

9 

16 

11 

6.00000 

2.64575 

0.60858 

10 

2 

9 

16 

12 

7.11111 

2.52212 

0.57735 

10 

5 

9 

16 

13 

7.33333 

2.54951 

0.57735 

10 

5 

9 

16 

14 

5.33333 

2.54951 

0.57735 

9 

0 

9 

16 

15 

7.66667 

2.78388 

0.65734 

10 

4 

9 

16 

16 

6.88889 

3.48010 

0.62854 

10 

0 

9 

16 

17 

3.33333 

4.15331 

0.50918 

10 

0 

9 

16 

18 

7.55556 

2.50555 

0.57735 

10 

5 

9 

16 

19 

7.66667 

2.54951 

0.60858 

10 

5 

9 

16 

20 

6.55556 

2.18581 

0.57735 

10 

5 

9 

Weighted  Combined  Service  Class  Score  for  Class  16: 

Score: 6. 56234  Sigma: 0 . 85603  Rho: 0.60206  evaluators : 9 .00000 


E-32 


NAVSWC  TR  90-246 


CANDIDATE:  MACH 

Results  with  data  for  evaluations  with  no  less  than  7  candidates  evaluated. 
220  files  for  candidate  MACH 


for  Class  0  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

0 

1 

8.40000 

1.72901 

0.40927 

10 

5 

20 

0 

2 

4.40000 

2.58335 

0.47697 

9 

0 

20 

0 

3 

6.75000 

1.58529 

0.43301 

10 

5 

20 

0 

4 

6.05000 

2.01246 

0.44441 

10 

3 

20 

0 

5 

6.05000 

2.18789 

0.45552 

9 

0 

20 

0 

6 

6.80000 

1.82382 

0.48218 

10 

3 

20 

0 

7 

5.55000 

1.90498 

0.45000 

8 

2 

20 

0 

8 

5.60000 

2.85436 

0.52202 

10 

0 

20 

Mean  Service  Class  Score  for  Class  0: 

Score: 6. 20000  Sigma: 0.75121  Rho: 0.16274  evaluators: 20. 00000 
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for  Class  1  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

1 

1 

6.13333 

2.46429 

0.29979 

10 

0 

45 

1 

2 

7.22222 

2.71267 

0.31190 

10 

0 

45 

1 

3 

5.91111 

1.92852 

0.29731 

10 

2 

45 

1 

4 

7.66667 

2.04495 

0.28803 

10 

4 

45 

1 

5 

7.44444 

2.58101 

0.31032 

10 

0 

45 

1 

6 

7.68889 

2.07608 

0.28545 

10 

3 

45 

1 

7 

7.26667 

2.15744 

0.30872 

10 

3 

45 

1 

8 

6.91111 

2.40097 

0.31817 

10 

2 

45 

1 

9 

3.86667 

3.35478 

0.30470 

10 

0 

45 

1 

10 

3.40000 

3.89872 

0.33555 

10 

0 

45 

1 

11 

4.60000 

2.70857 

0.29397 

10 

0 

45 

1 

12 

6.57778 

2.76742 

0.36175 

10 

0 

45 

1 

13 

6.37778 

3.01729 

0.33628 

10 

0 

45 

1 

14 

6.93333 

2.63197 

0.36107 

10 

0 

45 

1 

15 

6.46667 

2.23200 

0.30712 

10 

3 

45 

1 

16 

6.53333 

2.35102 

0.32886 

10 

1 

45 

1 

17 

6.80000 

2.64231 

0.33555 

10 

0 

45 

1 

18 

7.22222 

2.60148 

0.32660 

10 

0 

45 

1 

19 

5.11111 

2.92499 

0.34925 

10 

0 

45 

1 

20 

4.60000 

3.46016 

0.31111 

10 

0 

45 

1 

21 

3.51111 

2.87325 

0.34570 

8 

0 

45 

1 

22 

4.60000 

3.44040 

0.30307 

10 

0 

45 

1 

23 

3.84444 

3.45724 

0.32356 

10 

0 

45 

1 

24 

5.91111 

3.41669 

0.30307 

10 

0 

45 

1 

25 

5.68889 

3.73450 

0.32203 

10 

0 

45 

1 

26 

4 . 64444 

2.69811 

0.34783 

10 

0 

45 

1 

27 

5.15556 

2.89949 

0.34570 

10 

0 

45 

1 

28 

3.17778 

2.97939 

0.33481 

10 

0 

45 

1 

29 

2.55556 

2.83289 

0.32126 

10 

0 

45 

Mean  Service  Class  Score  for  Class  1: 

Score:5. 64908  Sigma:0. 68377  Rho:0. 43453  evaluators: 45. 00000 


for  Class  2  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

2  1  3.66667  3.39116  0.65734  909 

Weighted  Combined  Service  Class  Score  for  Class  2: 

Score : 3 . 66667  Sigma: 3. 39116  Rho: 0.65734  evaluators: 9. 00000 
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for  Class  3  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

3 

1 

7.71429 

3.68394 

0.60609 

10 

0 

7 

3 

2 

9.28571 

0.95119 

0.57143 

10 

8 

7 

3 

3 

7.28571 

3.59232 

0.51508 

10 

0 

7 

3 

4 

7.57143 

3.55233 

0.57143 

10 

0 

7 

3 

5 

6.42857 

3.86683 

0.69985 

10 

0 

7 

3 

6 

8.85714 

1.86445 

0.57143 

10 

5 

7 

3 

7 

7.42857 

3.73529 

0.51508 

10 

0 

7 

3 

8 

7.28571 

3.68394 

0.65465 

10 

0 

7 

3 

9 

5.71429 

4.02965 

0.69985 

10 

0 

7 

3 

10 

5.00000 

4.58258 

0.65465 

10 

0 

7 

3 

11 

8.85714 

1.86445 

0.51508 

10 

5 

7 

3 

12 

5.71429 

3.77334 

0.65465 

10 

0 

7 

3 

13 

7.42857 

3.73529 

0.60609 

10 

0 

7 

3 

14 

8.42857 

1.71825 

0.51508 

10 

5 

7 

3 

15 

7.42857 

3.73529 

0.51508 

10 

0 

7 

3 

16 

7.28571 

3.72891 

0.65465 

10 

0 

7 

3 

17 

5.00000 

4.47214 

0.65465 

10 

0 

7 

3 

18 

5.28571 

3.90360 

0.65465 

10 

0 

7 

3 

19 

7.28571 

2.56348 

0.65465 

10 

4 

7 

3 

20 

5.85714 

5.17779 

0.60609 

10 

0 

7 

3 

21 

7.00000 

3.55903 

0.65465 

10 

0 

7 

3 

22 

7.28571 

3.72891 

0.60609 

10 

0 

7 

3 

23 

8.71429 

1.97605 

0.60609 

10 

5 

7 

3 

24 

5.57143 

4.19750 

0.65465 

10 

0 

7 

Weighted  Combined  Service  Class  Score  for  Class  3: 

Score:7. 31408  Sigma:0. 75276  Rho:0. 13404  evaluators : 7 .00000 

for  Class  4  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

4  1  2.44444  3.53946  0.59835  809 

Weighted  Combined  Service  Class  Score  for  Class  4: 

Score:2. 44444  Sigma: 3 . 53946  Rho:0. 59835  evaluators: 9. 00000 


for  Class  5  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

5 

1 

3.84615 

2.37508 

0.57564 

10 

0 

13 

5 

2 

3.69231 

3.27579 

0.57564 

10 

0 

13 

5 

3 

1.69231 

2.21302 

0.58583 

5 

0 

13 

5 

4 

0.46154 

1.12660 

0.54393 

3 

0 

13 

5 

5 

0.61538 

1.55662 

0.56001 

5 

0 

13 

5 

6 

0.61538 

1.55662 

0.56001 

5 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  5: 

Score: 1.84597  Sigma: 0.93909  Rho: 0.38561  evaluators: 13. 00000 
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for  Class  6  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

6 

1 

0.87500 

2.47487 

0.68465 

7 

0 

8 

6 

2 

5.00000 

4.75094 

0.77055 

10 

0 

8 

6 

3 

2.12500 

4.01559 

0.77055 

10 

0 

8 

6 

4 

3.25000 

4.36708 

0.77055 

10 

0 

8 

6 

5 

0.62500 

1.76777 

0.71807 

5 

0 

8 

6 

6 

0.62500 

1.76777 

0.68465 

5 

0 

8 

6 

7 

5.37500 

5.04090 

0.68465 

10 

0 

8 

6 

8 

5.37500 

5.04090 

0.68465 

10 

0 

8 

6 

9 

5.37500 

5.04090 

0.68465 

10 

0 

8 

6 

10 

5.62500 

4.95516 

0.68465 

10 

0 

8 

6 

11 

5.62500 

4.95516 

0.68465 

10 

0 

8 

6 

12 

5.25000 

4.68280 

0.68465 

10 

0 

8 

6 

13 

5.25000 

4.68280 

0.68465 

10 

0 

8 

6 

14 

5.25000 

4.68280 

0.68465 

10 

0 

8 

6 

15 

0.00000 

0.00000 

0.68465 

0 

0 

8 

6 

16 

5.25000 

4.68280 

0.68465 

10 

0 

8 

6 

17 

5.25000 

4.68280 

0.68465 

10 

0 

8 

6 

18 

5.25000 

4.68280 

0.68465 

10 

0 

8 

6 

19 

5.62500 

4.95516 

0.68465 

10 

0 

8 

6 

20 

0.87500 

2.47487 

0.68465 

7 

0 

8 

Weighted  Combined  Service  Class  Score  for  Class  6: 

Score: 4. 12887  Sigma: 1.04928  Rho: 0.33975  evaluators: 8. 00000 


for  Class  7  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

7 

1 

2.37500 

2.66927 

0.82916 

7 

0 

8 

7 

2 

1.00000 

1.51186 

0.63738 

4 

0 

8 

7 

3 

1.00000 

1.51186 

0.63738 

4 

0 

8 

7 

4 

1.87500 

2.85044 

0.69597 

8 

0 

8 

7 

5 

1.87500 

2.85044 

0.69597 

8 

0 

8 

7 

6 

1.37500 

2.19984 

0.63738 

6 

0 

8 

7 

7 

1.00000 

1.51186 

0.63738 

4 

0 

8 

7 

8 

2.00000 

3.25137 

0.59948 

9 

0 

8 

7 

9 

3.12500 

3.44083 

0.75000 

10 

0 

8 

7 

10 

0.75000 

1.48805 

0.59948 

4 

0 

8 

7 

11 

0.75000 

1.48805 

0.55902 

4 

0 

8 

Weighted  Combined  Service  Class  Score  for  Class  7: 

Score: 1.57979  Sigma: 0.73932  Rho: 0.24992  evaluators: 8, 00000 
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for 

Class 

8  Mean  scores  for  each  Criterion  are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

8 

1 

2.90909 

2.21154 

0.64282 

5 

0 

11 

8 

2 

5.54545 

2 . 54416 

0.62984 

10 

2 

11 

8 

3 

3.63636 

4.52267 

0.56040 

10 

0 

11 

8 

4 

2.54545 

3.35749 

0.63636 

10 

0 

11 

8 

5 

2.27273 

3.43776 

0.63636 

10 

0 

11 

8 

6 

6.36364 

3.77552 

0.58210 

10 

0 

11 

8 

7 

6.54545 

2.29624 

0.58916 

10 

5 

11 

8 

8 

4.45455 

1.86353 

0.66183 

7 

0 

11 

8 

9 

1.27273 

2.00454 

0.63636 

5 

0 

11 

8 

10 

2.00000 

3.31662 

0.63636 

10 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  8: 

Score: 3. 81598  Sigma: 1.14258  Rho: 0.61725  evaluators: 11. 00000 


for  Class  9  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

9 

1 

8.88889 

1.96497 

0.56656 

10 

4 

9 

9 

2 

9.44444 

1.33333 

0.50918 

10 

6 

9 

9 

3 

7.88889 

2.14735 

0.59835 

10 

5 

9 

9 

4 

5.88889 

2.93447 

0.56656 

10 

0 

9 

9 

5 

9.00000 

1.73205 

0.47140 

10 

6 

9 

■9 

6 

9.44444 

1.33333 

0.47140 

10 

6 

9 

9 

7 

3.11111 

3.25747 

0.64788 

9 

0 

9 

9 

8 

7.55556 

2.35112 

0.50918 

10 

4 

9 

9 

9 

8.66667 

1.58114 

0.54433 

10 

6 

9 

9 

10 

5.77778 

3.30824 

0.64788 

10 

0 

9 

9 

11 

7.11111 

3.29562 

0.56656 

10 

0 

9 

9 

12 

8.55556 

2.18581 

0.56656 

10 

4 

9 

9 

13 

4.88889 

3.98260 

0.56656 

10 

0 

9 

9 

14 

7.66667 

3.84057 

0,56656 

10 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  9: 

Score: 7. 53155  Sigma: 0.99287  Rho: 0.73735  evaluators: 9. 00000 


for  Class  10  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

10  1  3.80000  3.73571  0.54772  8  0  10 

10  2  1.80000  2.57337  0.66332  8  0  10 

Weighted  Combined  Service  Class  Score  for  Class  10: 

Score: 2. 81191  Sigma: 2. 55076  Rho: 1.22544  evaluators: 10. 00000 
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for 

Class 

11  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

11 

1 

0.83333 

1.74946 

0.58333 

5 

0 

12 

11 

2 

0.33333 

1.15470 

0.61237 

4 

0 

12 

11 

3 

1.25000 

2.59808 

0.61237 

8 

0 

12 

11 

4 

1.25000 

2.59808 

0.61237 

8 

0 

12 

11 

5 

0.58333 

1.24011 

0.60093 

4 

0 

12 

11 

6 

0.00000 

0.00000 

0.60093 

0 

0 

12 

11 

7 

0.25000 

0.86603 

0.60093 

3 

0 

12 

11 

8 

0.08333 

0.28868 

0.60093 

1 

0 

12 

11 

9 

0.66667 

1.23091 

0.58333 

3 

0 

12 

11 

10 

1.33333 

1.77525 

0.62915 

4 

0 

12 

11 

11 

1.91667 

2.84312 

0.64010 

8 

0 

12 

11 

12 

0.66667 

1.23091 

0.66667 

3 

0 

12 

11 

13 

2.16667 

2.65718 

0.70711 

7 

0 

12 

11 

14 

0.75000 

1.60255 

0.58333 

5 

0 

12 

Weighted  Combined  Service  Class  Score  for  Class  11: 

Score: 0.83830  Sigma: 0.47485  Rho: 0.20289  evaluators: 12. 00000 


for 

Class 

12  Mean  i 

scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

12 

1 

8.18182 

1.94001 

0.58210 

10 

4 

11 

12 

2 

6.54545 

3.98406 

0.59613 

10 

0 

11 

12 

3 

9.45455 

0.82020 

0.51426 

10 

8 

11 

12 

4 

9.36364 

0.92442 

0.53783 

10 

8 

11 

12 

5 

7.72727 

2.96954 

0.58210 

10 

0 

11 

12 

6 

3.45455 

2.76997 

0.68030 

7 

0 

11 

12 

7 

2.00000 

2.60768 

0.71002 

7 

0 

11 

12 

8 

1.72727 

2.64919 

0.71002 

7 

0 

11 

12 

9 

3.27273 

3.03615 

0.71002 

7 

0 

11 

12 

10 

3.27273 

3.03615 

0.71002 

7 

0 

11 

12 

11 

4.18182 

3.60051 

0.64922 

9 

0 

11 

12 

12 

3.72727 

2.49363 

0.64922 

7 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  12: 

Score:5. 43678  Sigma:0. 97831  Rho:0. 61897  evaluators: 11.00000 
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for  Class  13  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

13 

1 

5.63636 

4.22546 

0.66804 

10 

0 

11 

13 

2 

6.00000 

3.68782 

0.61658 

10 

0 

11 

13 

3 

6.72727 

3.22772 

0.60302 

10 

0 

11 

13 

4 

3.09091 

3.04810 

0.65555 

8 

0 

11 

13 

5 

2.81818 

3.65563 

0.65555 

10 

0 

11 

13 

6 

3.18182 

3.28080 

0.67420 

10 

0 

11 

13 

7 

3.00000 

3.00000 

0.68635 

9 

0 

11 

13 

8 

3.09091 

3.91036 

0.66804 

10 

0 

11 

13 

9 

2.18182 

3.25017 

0.66804 

10 

0 

11 

13 

10 

2.90909 

3.61814 

0.66804 

10 

0 

11 

13 

11 

2.72727 

2.93567 

0.60302 

8 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  13: 

Score:3. 78866  Sigma:l. 13433  Rho:0. 41706  evaluators: 11. 00000 


for  Class  14  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

14 

1 

5.11111 

4.25572 

0.70273 

10 

0 

9 

14 

2 

6.22222 

3.19287 

0.70273 

10 

0 

9 

14 

3 

4.22222 

4.65773 

0.76980 

10 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  14: 
Score:5. 26021  Sigma:2. 44778  Rho:0. 89659  evaluators : 9 .00000 


for 

Class 

15  Mean  scores  for 

each  Criterion 

ire: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

15 

1 

2.53846 

3.01705 

0.56527 

8 

0 

13 

15 

2 

2.00000 

2.76887 

0.56527 

8 

0 

13 

15 

3 

1.15385 

2.26738 

0.58076 

7 

0 

13 

15 

4 

1.07692 

2.28989 

0.56527 

7 

0 

13 

15 

5 

1.07692 

2.28989 

0.56527 

7 

0 

13 

15 

6 

2.30769 

2.78043 

0.60569 

7 

0 

13 

15 

7 

1.76923 

2.24179 

0.59584 

5 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  15: 

Score: 1.79347  Sigma: 1.03047  Rho: 0.30390  evaluators: 13. 00000 
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for 

Class 

16  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

16 

1 

6.00000 

3.04138 

0.59835 

10 

0 

9 

16 

2 

6.11111 

3.10018 

0.59835 

10 

0 

9 

16 

3 

6.55556 

2.60342 

0.56656 

10 

4 

9 

16 

4 

6.55556 

2.60342 

0.56656 

10 

4 

9 

16 

5 

5.77778 

3.03223 

0.56656 

10 

0 

9 

16 

6 

5.55556 

3.00463 

0.64788 

10 

0 

9 

16 

7 

4.77778 

3.59784 

0.64788 

10 

0 

9 

16 

8 

4.22222 

3.92994 

0.64788 

10 

0 

9 

16 

9 

6.00000 

2.59808 

0.59835 

10 

2 

9 

16 

10 

5.33333 

3.67423 

0.64788 

10 

0 

9 

16 

11 

3.88889 

4.16667 

0.66667 

10 

0 

9 

16 

12 

3.88889 

3.33333 

0.64788 

10 

0 

9 

16 

13 

5.00D00 

3.53553 

0.72008 

10 

0 

9 

16 

14 

3.55556 

3.71184 

0.72008 

10 

0 

9 

16 

15 

6.44444 

2.12786 

0.59835 

10 

5  ' 

9 

16 

16 

3.55556 

3.35824 

0.76174 

10 

0 

9 

16 

17 

3.00000 

3.50000 

0.76174 

10 

0 

9 

16 

18 

2.77778 

3.63242 

0.76174 

10 

0 

9 

16 

19 

4.66667 

3.12250 

0.72008 

10 

0 

9 

16 

20 

5.11111 

3.21887 

0.67586 

10 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  16: 
Score:4. 98215  Sigraa:0. 86838  Rho:0. 47406  evaluators: 9. 00000 
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CANDIDATE:  ORKID 


Results  with  data  for  evaluations  with  no  less  than  7  candidates  evaluated. 
218  files  for  candidate  ORKID 

for  Class  0  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

0 

1 

7.90000 

2.88189 

0.41833 

10 

0 

20 

0 

2 

5.60000 

2.72223 

0.48734 

10 

0 

20 

0 

3 

4.05000 

2.35025 

0.49497 

10 

0 

20 

0 

4 

6.40000 

2.28035 

0.48218 

10 

0 

20 

0 

5 

5.05000 

1.95946 

0.49497 

8 

0 

20 

0 

6 

6.95000 

2.11449 

0.47958 

10 

5 

20 

0 

7 

7.50000 

2.43872 

0.49497 

10 

0 

20 

0 

8 

6.05000 

2.48098 

0.53852 

10 

0 

20 

Mean  Service  Class  Score  for  Class  0: 

Score:6. 18750  Sigma: 0.85561  Rho:0. 17230  evaluators: 20. 00000 
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CANDIDATE:  ORKID 


for  Class  1  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

1 

1 

5.24444 

1.33409 

0.27756 

9 

3 

45 

1 

2 

6.66667 

1.89497 

0.28974 

10 

3 

45 

1 

3 

7.28889 

1.84172 

0.29565 

10 

5 

45 

1 

4 

7.00000 

2.43086 

0.28197 

10 

1 

45 

1 

5 

7.71111 

2.37048 

0.27397 

10 

0 

45 

1 

6 

6.04444 

2.23562 

0.29397 

10 

0 

45 

1 

7 

7.88889 

2.37623 

0.27035 

10 

0 

45 

1 

8 

6.91111 

2.28456 

0.28284 

10 

0 

45 

1 

9 

6.97778 

3.01126 

0.28021 

10 

0 

45 

1 

10 

0.31111 

0.94922 

0.24444 

5 

0 

45 

1 

11 

3.93333 

2.28035 

0.27035 

8 

0 

45 

1 

12 

5.31111 

3.38326 

0.33775 

10 

0 

45 

1 

13 

5.15556 

3.01478 

0.32735 

10 

0 

45 

1 

14 

6.66667 

3.42451 

0.33921 

10 

0 

45 

1 

15 

6.97778 

2.34025 

0.28545 

10 

0 

45 

1 

16 

8.11111 

1.86136 

0.29897 

10 

2 

45 

1 

17 

7.20000 

2.60768 

0.30952 

10 

0 

45 

1 

18 

8.26667 

2.06045 

0.28889 

10 

0 

45 

1 

19 

5.55556 

3.04180 

0.34427 

10 

0 

45 

1 

20 

7.26667 

3.01813 

0.29481 

10 

0 

45 

1 

21 

2.33333 

2.92326 

0.32126 

9 

0 

45 

1 

22 

5.22222 

3.24660 

0.30144 

10 

0 

45 

1 

23 

2.35556 

2.26791 

0.27844 

10 

0 

45 

1 

24 

2.44444 

2.28190 

0.27487 

8 

0 

45 

1 

25 

6.88889 

2.87799 

0.29313 

10 

0 

45 

1 

26 

4.75556 

3.39890 

0.32660 

10 

0 

45 

1 

27 

4.57778 

3.26475 

0.32049 

10 

0 

45 

1 

28 

2.17778 

2.44288 

0.29313 

10 

0 

45 

1 

29 

1.04444 

1.65083 

0.28803 

6 

0 

45 

Mean  Service  Class  Score  for  Class  1: 

Score : 5 .45714  Sigma: 0.64873  Rho:0. 44215  evaluators: 45. 00000 


for  Class  2  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

2  1  2.44444  3.12694  0.65734  909 

Weighted  Combined  Service  Class  Score  for  Class  2: 

Score : 2 .44444  Sigma:3. 12694  Rho:0. 65734  evaluators: 9. 00000 
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for  Class  3  Mean  scores  for  each  Criterion  are; 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

3 

1 

0.42857 

1.13389 

0.45175 

3 

0 

7 

3 

2 

0.42857 

1.13389 

0.45175 

3 

0 

7 

3 

3 

0.42857 

1.13389 

0.45175 

3 

0 

7 

3 

4 

0.42857 

1.13389 

0.45175 

3 

0 

7 

3 

5 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

6 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

7 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

8 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

9 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

10 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

11 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

12 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

13 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

14 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

15 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

16 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

17 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

18 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

19 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

20 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

21 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

22 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

23 

0.00000 

0.00000 

0.45175 

0 

0 

7 

3 

24 

0.00000 

0.00000 

0.51508 

0 

0 

7 

Weighted  Combined  Service  Class  Score  for  Class  3: 

Score: 0.08508  Sigma: 0.11401  Rho: 0.10093  evaluators: 7. 00000 


for  Class  4  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

4  1  0.77778  1.71594  0.50918  5  09 

Weighted  Combined  Service  Class  Score  for  Class  4: 

Score: 0.77778  Sigma: 1.71594  Rho: 0.50918  evaluators: 9. 00000 


for  Class  5  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

5 

1 

3.76923 

1.92154 

0.42829 

7 

0 

13 

5 

2 

3.53846 

2.47034 

0.49852 

9 

0 

13 

5 

3 

2.61538 

2.87340 

0.51602 

8 

0 

13 

5 

4 

0.69231 

1.18213 

0.42829 

3 

0 

13 

5 

5 

3.69231 

3.01066 

0.46790 

8 

0 

13 

5 

6 

0.92308 

1.70595 

0.44853 

5 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  5: 

Score: 2.55932  Sigma: 1.04134  Rho: 0.47150  evaluators: 13. 00000 
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for  Class  6  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

6 

1 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

2 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

3 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

4 

0.00000 

0.00000 

0.51539 

0 

0 

8 

6 

5 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

6 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

7 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

8 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

9 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

10 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

11 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

12 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

13 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

14 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

15 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

16 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

17 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

18 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

19 

0.00000 

0.00000 

0.46771 

0 

0 

8 

6 

20 

0.00000 

0.00000 

0.46771 

0 

0 

8 

Weighted  Combined  Service  Class  Score  for  Class  6: 

Score: 0.00000  Sigma: 0.00000  Rho:C. 10620  evaluators: 8. 00000 


for  Class  7  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

7 

1 

1.12500 

1.64208 

0.66144 

4 

0 

8 

7 

2 

0.50000 

0.92582 

0.62500 

2 

0 

8 

7 

3 

0.50000 

0.92582 

0.62500 

2 

0 

8 

7 

4 

0.50000 

0.92582 

0.62500 

2 

0 

8 

7 

5 

0.50000 

0.92582 

0.62500 

2 

0 

8 

7 

6 

1.00000 

1.51186 

0.66144 

4 

0 

8 

7 

7 

0.37500 

0.74402 

0.62500 

2 

0 

8 

7 

8 

0.25000 

0.70711 

0.62500 

2 

0 

8 

7 

9 

0.50000 

0.92582 

0.66144 

2 

0 

8 

7 

10 

0.50000 

0.92582 

0.62500 

2 

0 

8 

7 

11 

0.50000 

0.92582 

0.62500 

2 

0 

8 

Weighted  Combined  Service  Class  Score  for  Class  7: 

Score: 0.57940  Sigma: 0.32990  Rho: 0.20004  evaluators : 8 .00000 
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for  Class  8  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

8 

1 

1.18182 

1.83402 

0.37483 

5 

0 

11 

8 

2 

1.45455 

1.75292 

0.48105 

5 

0 

11 

8 

3 

0.27273 

0.90453 

0.48105 

3 

0 

11 

8 

4 

0.09091 

0.30151 

0.43598 

1 

0 

11 

8 

5 

0.45455 

1.50756 

0.48105 

5 

0 

11 

8 

6 

0.00000 

0.00000 

0.43598 

0 

0 

11 

8 

7 

0.54545 

1.50756 

0.43598 

5 

0 

11 

8 

8 

0.45455 

1.50756 

0.46355 

5 

0 

11 

8 

9 

0.00000 

0.00000 

0.43598 

0 

0 

11 

8 

10 

0.00000 

0.00000 

0.43598 

0 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  8: 

Score:0. 49861  Sigma:0. 42480  Rho:0. 16782  evaluators: 11. 00000 


for  Class  9  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

9 

1 

8.88889 

0.92796 

0.43033 

10 

8 

9 

9 

2 

5.22222 

1.09291 

0.47140 

8 

4 

9 

9 

3 

9.77778 

0.44096 

0.43033 

10 

9 

9 

9 

4 

4.55556 

1.74005 

0.47140 

6 

0 

9 

9 

5 

9.55556 

0.72648 

0.43033 

10 

8 

9 

9 

6 

9.66667 

0.70711 

0.43033 

10 

8 

9 

9 

7 

6.00000 

2.95804 

0.50918 

10 

0 

9 

9 

8 

7.88889 

2.31541 

0.47140 

10 

5 

9 

9 

9 

3.33333 

4.03113 

0.56656 

9 

0 

9 

9 

10 

5.55556 

2.40370 

0.47140 

9 

2 

9 

9 

11 

2.55556 

3.32081 

0.59835 

8 

0 

9 

9 

12 

8.00000 

1.87083 

0.47140 

10 

5 

9 

9 

13 

2.22222 

1.64148 

0.49690 

5 

0 

9 

9 

14 

0.44444 

1.33333 

0.50918 

4 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  9: 

Score: 6. 25949  Sigma: 0.83393  Rho: 0.64399  evaluators: 9. 00000 


for  Class  10  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

10 

1 

0.20000 

0.63246 

0.46904 

2 

0 

10 

10 

2 

0.40000 

1.26491 

0.51962 

4 

0 

10 

Weighted  Combined  Service  Class  Score  for  Class  10: 

Score:0. 29881  Sigma:0. 71131  Rho:0. 36778  evaluators: 10. 00000 
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CANDIDATE:  ORKID 


for  Class  11  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

11 

1 

0.16667 

0.38925 

0.45644 

1 

0 

12 

11 

2 

0.16667 

0.38925 

0.45644 

1 

0 

12 

11 

3 

0.00000 

0.00000 

0.43301 

0 

0 

12 

11 

4 

0.00000 

0.00000 

0.43301 

0 

0 

12 

11 

5 

0.58333 

1.24011 

0.45644 

4 

0 

12 

11 

6 

0.00000 

0.00000 

0.43301 

0 

0 

12 

11 

7 

0.08333 

0.28868 

0.47871 

1 

0 

12 

11 

8 

0.33333 

0.77850 

0.45644 

2 

0 

12 

11 

9 

0.00000 

0.00000 

0.43301 

0 

0 

12 

11 

10 

0.25000 

0.62158 

0.45644 

2 

0 

12 

11 

11 

0.08333 

0.28868 

0.43301 

1 

0 

12 

11 

12 

0.16667 

0.38925 

0.43301 

1 

0 

12 

11 

13 

0.83333 

1.94625 

0.47871 

5 

0 

12 

11 

14 

0.00000 

0.00000 

0.43301 

0 

0 

12 

Weighted  Combined  Service  Class  Score  for  Class  11: 

Score: 0.18688  Sigma: 0.17870  Rho: 0.12490  evaluators: 12. 00000 


for 

Class 

12  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

12 

1 

0.18182 

0.60302 

0.46355 

2 

0 

11 

12 

2 

0.18182 

0.60302 

0.46355 

2 

0 

11 

12 

3 

7.09091 

3.53425 

0.58916 

10 

0 

11 

12 

4 

0.36364 

1.20605 

0.46355 

4 

0 

11 

12 

5 

3.90909 

3.53425 

0.57496 

10 

0 

11 

12 

6 

0.54545 

1.50756 

0.50616 

5 

0 

11 

12 

7 

0.72727 

1.55505 

0.46355 

5 

0 

11 

12 

8 

0.72727 

1.55505 

0.46355 

5 

0 

11 

12 

9 

1.72727 

2.14900 

0.51426 

6 

0 

11 

12 

10 

1.72727 

2.14900 

0.51426 

6 

0 

11 

12 

11 

1.18182 

2.63887 

0.46355 

7 

0 

11 

12 

12 

1.45455 

2.65946 

0.46355 

8 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  12: 

Score: 1.74980  Sigma: 0.68857  Rho: 0,26224  evaluators: 11. 00000 
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CANDIDATE:  ORKID 


for 

Class 

13  Mean  scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

13 

1 

7.18182 

2.18258 

0.48105 

10 

5 

11 

13 

2 

8.63636 

1.96330 

0.40656 

10 

5 

11 

13 

3 

0.27273 

0.90453 

0.37483 

3 

0 

11 

13 

4 

6.63636 

3.35478 

0.43598 

10 

1 

11 

13 

5 

0.36364 

1.20605 

0.37483 

4 

0 

11 

13 

6 

6.54545 

2.94495 

0.40656 

10 

0 

11 

13 

7 

4.81818 

3.12468 

0.53009 

10 

0 

11 

13 

8 

3.18182 

2.78633 

0.48105 

9 

0 

11 

13 

9 

3.81818 

4.04520 

0.48105 

10 

0 

11 

13 

10 

6.72727 

2.76011 

0.40656 

10 

2 

11 

13 

11 

4.54545 

2.87623 

0.48105 

9 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  13: 

Score: 5. 15853  Sigma: 0.98554  Rho: 0.54480  evaluators: 11. 00000 


for  Class  14  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

14 

1 

0.00000 

0.00000 

0.49690 

0 

0 

9 

14 

2 

1.66667 

3.53553 

0.43033 

10 

0 

9 

14 

3 

0.33333 

1.00000 

0.49690 

3 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  14: 

Score: 0.69581  Sigma: 1.31501  Rho: 0.31107  evaluators : 9 . 00000 


for  Class  15  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

15 

1 

5.76923 

1.23517 

0.42133 

8 

5 

13 

15 

2 

7.23077 

2.89119 

0.38462 

10 

0 

13 

15 

3 

0.61538 

1.55662 

0.38462 

5 

0 

13 

15 

4 

0.23077 

0.83205 

0.38462 

3 

0 

13 

15 

5 

0.23077 

0.83205 

0.38462 

3 

0 

13 

15 

6 

6.92308 

3.25222 

0.38462 

10. 

0 

13 

15 

7 

1.61538 

2.10311 

0.47419 

5 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  15: 

Score:3. 68936  Sigma:0. 95031  Rho:0. 48124  evaluators: 13. 00000 
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CANDIDATE:  ORKID 


for  Class  16  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

16 

1 

4.22222 

2.48886 

0.47140 

7 

0 

9 

16 

2 

3.55556 

2.24227 

0.54433 

5 

0 

9 

16 

3 

4.77778 

3.59784 

0.56656 

10 

0 

9 

16 

4 

4.77778 

3.59784 

0.56656 

10 

0 

9 

16 

5 

4.00000 

3.04138 

0.56656 

10 

0 

9 

16 

6 

4.55556 

3.28295 

0.56656 

10 

0 

9 

16 

7 

3.11111 

2.42097 

0.50918 

5 

0 

9 

16 

8 

3.11111 

2.42097 

0.50918 

5 

0 

9 

16 

9 

3.11111 

2.66667 

0.50918 

7 

0 

9 

16 

10 

1.55556 

2.35112 

0.43033 

5 

0 

9 

16 

11 

3.11111 

3.25747 

0.47140 

9 

0 

9 

16 

12 

5.88889 

3.98260 

0.43033 

10 

0 

9 

16 

13 

5.00000 

3.53553 

0.47140 

10 

0 

9 

16 

14 

2.77778 

2.81859 

.0.47140 

7 

0 

9 

16 

15 

4.66667 

4.09268 

0.43033 

10 

0 

9 

16 

16 

3.00000 

3.42783 

0.53287 

10 

0 

9 

16 

17 

2.88889 

3.40751 

0.53287 

10 

0 

9 

16 

18 

1.77778 

2.22361 

0.53287 

5 

0 

9 

16 

19 

3.66667 

2.64575 

0.56656 

8 

0 

9 

16 

20 

2.00000 

2.64575 

0.47140 

7 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  16: 

Score: 3. 62818  Sigma: 0.76749  Rho: 0.34834  evaluators: 9. 00000 
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CANDIDATE:  POSIX 

Results  with  data  for  evaluations  with  no  less  than  7  candidates  evaluated. 
217  files  for  candidate  POSIX 

for  Class  0  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

0 

1 

9.45000 

2.23548 

0.26926 

10 

0 

20 

0 

2 

6.05000 

2.03845 

0.40927 

10 

2 

20 

0 

3 

5.95000 

2.25890 

0.44721 

9 

0 

20 

0 

4 

7.95000 

1.70062 

0.38079 

10 

5 

20 

0 

5 

8.75000 

2.29129 

0.38079 

10 

0 

20 

0 

6 

6.40000 

2.54227 

0.45277 

10 

0 

20 

0 

7 

8.85000 

1.34849 

0.37081 

10 

5 

20 

0 

8 

6.80000 

2.33057 

0.45277 

9 

0 

20 

Mean  Service  Class  Score  for  Class  0: 

Score:7. 52500  Sigma:0. 75115  Rho:0. 14128  evaluators: 20. 00000 
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CANDIDATE:  POSIX 


for  Class  1  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

1 

1 

7.35556 

1.81102 

0.27666 

10 

3 

45 

1 

2 

8.75556 

1.38425 

0.25915 

10 

5 

45 

1 

3 

6.55556 

1.90162 

0.26574 

10 

3 

45 

1 

4 

9.04444 

1.47641 

0.23413 

10 

4 

45 

1 

5 

7.06667 

2.53521 

0.27397 

10 

0 

45 

1 

6 

7.82222 

2.22883 

0.28371 

10 

0 

45 

1 

7 

7.08889 

2.85897 

0.28458 

10 

0 

45 

1 

8 

7.13333 

2.35102 

0.29481 

10 

0 

45 

1 

9 

5.37778 

2.67385 

0.27126 

10 

0 

45 

1 

10 

7.06667 

3.09985 

0.27126 

10 

0 

45 

1 

11 

7.44444 

2.76796 

0.24444 

10 

0 

45 

1 

12 

6.53333 

2.90454 

0.32126 

10 

0 

45 

1 

13 

5.71111 

2.75204 

0.29814 

10 

0 

45 

1 

14 

7.55556 

2.22134 

0.34427 

10 

-  0 

45 

1 

15 

7.97778 

2.09424 

0.28197 

10 

0 

45 

1 

16 

7.80000 

2.43647 

0.29731 

10 

0 

45 

1 

17 

7.95556 

2.37368 

0.30631 

10 

0 

45 

1 

18 

6.77778 

2.48531 

0.30631 

10 

0 

45 

1 

.  19 

7.53333 

2.84125 

0.31111 

10 

0 

45 

1 

20 

5.48889 

2.95898 

0.33110 

10 

0 

45 

1 

21 

5.53333 

2.84125 

0.30388 

10 

0 

45 

1 

22 

5.20000 

2.59019 

0.30551 

10 

0 

45 

1 

23 

3.77778 

3.22553 

0.29313 

10 

0 

45 

1 

24 

6.86667 

2.23200 

0.28803 

10 

1 

45 

1 

25 

5.68889 

2.41983 

0.33110 

10 

0 

45 

1 

26 

6.04444 

3.00723 

0.30551 

10 

0 

45 

1 

27 

5.35556 

2.59506 

0.33628 

10 

0 

45 

1 

28 

2.40000 

2.82360 

0.30952 

10 

0 

45 

1 

29 

2.04444 

2.55801 

0.31972 

9 

0 

45 

Mean  Service  Class  Score  for  Class  1: 

Score:6, 46189  Sigma:0. 68018  Rho:0. 49769  evaluators: 45. 00000 


for  Class  2  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

2  1  3.44444  4.30439  0.57735  10  0  9 

Weighted  Combined  Service  Class  Score  for  Class  2: 

Score: 3.44444  Sigma: 4. 30439  Rho: 0.57735  evaluators: 9. 00000 
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CANDIDATE:  POSIX 


for  Class  3  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

3 

1 

7.00000 

3.55903 

0.69985 

10 

1 

7 

3 

2 

7.85714 

2.41030 

0.65465 

10 

3 

7 

3 

3 

7.85714 

2.41030 

0.65465 

10 

3 

7 

3 

4 

7.28571 

3.72891 

0.65465 

10 

1 

7 

3 

5 

7.28571 

3.72891 

0.65465 

10 

1 

7 

3 

6 

8.57143 

2.57275 

0.65465 

10 

3 

7 

3 

7 

8.42857 

2.63674 

0.65465 

10 

3 

7 

3 

8 

7.14286 

2.79455 

0.69985 

10 

3 

7 

3 

9 

6.42857 

3.40867 

0.69985 

10 

1 

7 

3 

10 

6.71429 

3.54562 

0.69985 

10 

1 

7 

3 

11 

7.42857 

3.35942 

0.57143 

10 

1 

7 

3 

12 

6.28571 

3.40168 

0.69985 

10 

1 

7 

3 

13 

8.14286 

2.54484 

0.65465 

10 

3 

7 

3 

14 

7.28571 

2.56348 

0.65465 

10 

3 

7 

3 

15 

8.42857 

2.50713 

0.65465 

10 

3 

7 

3 

16 

8.14286 

2.41030 

0.65465 

10 

3 

7 

3 

17 

6.71429 

3.72891 

0.69985 

10 

1 

7 

3 

18 

6.85714 

3.53217 

0.65465 

10 

1 

7 

3 

19 

7.14286 

3.62531 

0.65465 

10 

1 

7 

3 

20 

6.28571 

4.34796 

0.57143 

10 

0 

7 

3 

21 

5.71429 

4.15188 

0.62270 

10 

0 

7 

3 

22 

7.28571 

3.72891 

0.65465 

10 

1 

7 

3 

23 

4.71429 

4.75094 

0.69985 

10 

0 

7 

3 

24 

5.00000 

4.69042 

0.62270 

10 

0 

7 

Weighted  Combined  Service  Class  Score  for  Class  3: 

Score: 7. 27836  Sigma: 0.72626  Rho: 0.14592  evaluators: 7. 00000 


for  Class  4  Mean  scores  for  each  Criterion  are: 

Cl  Cr  Score  Std  dev  Rho  Max  Min  #Evaluators 

4  1  1.44444  2.18581  0.54433  509 

Weighted  Combined  Service  Class  Score  for  Class  4: 

Score:l. 44444  Sigma: 2 . 18581  Rho:0. 54433  evaluators: 9. 00000 


for  Class  5  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

5 

1 

5.61538 

2.29269 

0.50442 

10 

2 

13 

5 

2 

5.84615 

2.57702 

0.53294 

10 

2 

13 

5 

3- 

5.92308 

2.53185 

0.60079 

10 

2 

13 

5 

4 

0.46154 

0.96742 

0.48650 

3 

0 

13 

5 

5 

1.53846 

2.33150 

0.61056 

7 

0 

13 

5 

6 

3.23077 

2.91987 

0.60079 

9 

0 

13 

Weighted  Combined  Service  Class  Score  for  Class  5: 

Score:3. 87614  Sigma:l. 15971  Rho:0. 64637  evaluators : 13 .00000 
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CANDIDATE:  POSIX 


for  Class  6  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

6 

1 

8.50000 

2.26779 

0.46771 

10 

5 

8 

6 

2 

4.75000 

2.76457 

0.64952 

10 

0 

8 

6 

3 

5.12500 

2.79987 

0.66144 

10 

0 

8 

6 

4 

7.75000 

2.37547 

0.66144 

10 

5 

8 

6 

5 

6.62500 

4.43807 

0.71807 

10 

0 

8 

6 

6 

2.87500 

2.74838 

0.62500 

8 

0 

8 

6 

7 

9.87500 

0.35355 

0.51539 

10 

9 

8 

6 

8 

9.25000 

1.75255 

0.58630 

10 

5 

8 

6 

9 

9.25000 

1.75255 

0.58630 

10 

5 

8 

6 

10 

10.00000 

0.00000 

0.51539 

10 

10 

8 

6 

11 

8.75000 

1.90863 

0.64952 

10 

5 

8 

6 

12 

10.00000 

0.00000 

0.54486 

10 

10 

8 

6 

13 

10.00000 

0.00000 

0.58630 

10 

10 

8 

6 

14 

9.37500 

1.76777 

0.54486 

10 

5 

8 

6 

15 

1.87500 

3.72012 

0.54486 

10 

0 

8 

6 

16 

10.00000 

0.00000 

0.46771 

10 

10 

8 

6 

17 

9.62500 

0.74402 

0.54486 

10 

8 

8 

6 

18 

9.50000 

1.06904 

0.64952 

10 

7 

8 

6 

19 

9.62500 

1.06066 

0.54486 

10 

7 

8 

6 

20 

6.25000 

5.17549 

0.58630 

10 

0 

8 

Weighted  Combined  Service  Class 

Score  for 

Class  6: 

Score: 8. 24423  Sigma: 0.74391  Rho: 0.58990  evaluators : 8 . 00000 


for  Class  7  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

7 

1 

2.37500 

3.06769 

0.66144 

7 

0 

8 

7 

2 

9.37500 

0.91613 

0.55902 

10 

8 

8 

7 

3 

9.50000 

0.92582 

0.55902 

10 

8 

8 

7 

4 

9.62500 

0.74402 

0.55902 

10 

8 

8 

7 

5 

9.62500 

0.74402 

0.55902 

10 

8 

8 

7 

6 

8.50000 

3.20713 

0.59948 

10 

1 

8 

7 

7 

7.87500 

2.29518 

0.63738 

10 

4 

8 

7 

8 

8.87500 

1.72689 

0.55902 

10 

5 

8 

7 

9 

6.50000 

4.24264 

0.59948 

10 

0 

8 

7 

10 

2.12500 

2.47487 

0.63738 

5 

0 

8 

7 

11 

0.62500 

1.76777 

0.59948 

5 

0 

8 

Weighted  Combined  Service  Class 

Score  for 

Class  7: 

Score: 7. 11975  Sigma: 0.91094  Rho: 0.63434  evaluators: 8. 00000 
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CANDIDATE:  POSIX 


for  Class  8  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

8 

1 

2.00000 

2.09762 

0.60302 

5 

0 

11 

8 

2 

2.54545 

2.84125 

0.54545 

8 

0 

11 

8 

3 

4.81818 

3.45885 

0.65555 

10 

0 

11 

8 

4 

3.90909 

3.96118 

0.67420 

10 

0 

11 

8 

5 

4.00000 

4.17133 

0.65555 

10 

0 

11 

8 

6 

3.54545 

3.69767 

0.73293 

10 

0 

11 

8 

7 

4.63636 

3.77552 

0.67420 

10 

0 

11 

8 

8 

2.27273 

2.64919 

0.62324 

7 

0 

11 

8 

9 

2.27273 

2.64919 

0.62324 

7 

0 

11 

8 

10 

1.54545 

2.46429 

0.65555 

7 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  8: 

Score: 3. 19931  Sigma: 1.13805  Rho: 0.51459  evaluators: 11. 00000 


for 

Class 

9  Mean  scores  for  each  Criterion  are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

9 

1 

6.00000 

3.42783 

0.47140 

10 

0 

9 

9 

2 

6.77778 

3.63242 

0.47140 

10 

0 

9 

9 

3 

4.88889 

3.25747 

0.47140 

10 

0 

9 

9 

4 

3.44444 

4.00347 

0.47140 

10 

0 

9 

9 

5 

6.44444 

3.50397 

0.47140 

10 

0 

9 

9 

6 

5.55556 

3.20590 

0.47140 

10 

0 

9 

9 

7 

8.66667 

2.12132 

0.43033 

10 

5 

9 

9 

8 

8.22222 

2.10819 

0.47140 

10 

4 

9 

9 

9 

6.44444 

3.35824 

0.47140 

10 

0 

9 

9 

10 

6.33333 

3.16228 

0.47140 

10 

0 

9 

9 

11 

5.22222 

2.77389 

0.47140 

10 

0 

9 

9 

12 

8.66667 

2.34521 

0.47140 

10 

3 

9 

9 

13 

0.33333 

1.00000 

0.43033 

3 

0 

9 

9 

14 

7.77778 

3.30824 

0.47140 

10 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  9: 

Score: 6. 21389  Sigma: 1.00233  Rho: 0.59398  evaluators: 9. 00000 


for  Class  10  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

10 

1 

0.40000 

0.84327 

0.54772 

2 

0 

10 

10 

2 

0.30000 

0.94868 

0.46904 

3 

0 

10 

Weighted  Combined  Service  Class  Score  for  Class  10: 

Score: 0.35060  Sigma: 0.64782  Rho: 0.38530  evaluators : 10. 00000 
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CANDIDATE:  POSIX 


for  Class  11  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

11 

1 

4.00000 

4.06761 

0.55277 

10 

0 

12 

11 

2 

4.08333 

4.23102 

0.58333 

10 

0 

12 

11 

3 

1.00000 

2.37410 

0.47871 

7 

0 

12 

11 

4 

0.41667 

1.44338 

0.53359 

5 

0 

12 

11 

5 

1.08333 

1.72986 

0.53359 

5 

0 

12 

11 

6 

0.25000 

0.86603 

0.54645 

3 

0 

12 

11 

7 

0.83333 

1.64225 

0.59512 

5 

0 

12 

11 

8 

2.50000 

1.78377 

0.63465 

5 

0 

12 

11 

9 

0.33333 

0.88763 

0.56519 

3 

0 

12 

11 

10 

1.16667 

1.58592 

0.62915 

5 

0 

12 

11 

11 

2.58333 

2.46644 

0.63465 

7 

0 

12 

11 

12 

2.25000 

1.91288 

0.66144 

5 

0 

12 

11 

13 

2.75000 

3.62128 

0.62915 

10 

0 

12 

11 

14 

1.08333 

2.93748 

0.57735 

10 

0 

12 

Weighted  Combined  Service  Class  Score  for  Class  11: 

Score:l. 77788  Sigma:0. 72157  Rho:0. 28572  evaluators: 12. 00000 


for 

Class 

12  Mean  i 

scores  for 

each  Criterion 

are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

12 

1 

1.36364 

2.11058 

0.53783 

7 

0 

11 

12 

2 

7.27273 

3.77070 

0.55298 

10 

0 

11 

12 

3 

3.54545 

2.69680 

0.57496 

8 

0 

11 

12 

4 

0.72727 

1.67874 

0.53009 

5 

0 

11 

12 

5 

8.63636 

1.50151 

0.53783 

10 

5 

11 

12 

6 

1.81818 

2.44206 

0.55298 

7 

0 

11 

12 

7 

3.27273 

2.19504 

0.72157 

7 

0 

11 

12 

8 

3.27273 

2.19504 

0.72157 

7 

0 

11 

12 

9 

3.81818 

2.04050 

0.69234 

7 

0 

11 

12 

10 

3.81818 

2.04050 

0.69234 

7 

0 

11 

12 

11 

4.36364 

2.20330 

0.62984 

8 

2 

11 

12 

12 

3.81818 

2.60070 

0.62984 

7 

0 

11 

Weighted  Combined  Service  Class  Score  for  Class  12: 

Score: 3. 85600  Sigma :0. 83191  Rho: 0.50407  evaluators: 11.00000 
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CANDIDATE:  POSIX 


for  Class  13  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

13 

1 

8.63636 

2.06265 

0.50616 

10 

5 

11 

13 

2 

8.09091 

2.21154 

0.46355 

10 

5 

11 

13 

3 

7.72727 

3.31936 

0.50616 

10 

0 

11 

13 

4 

9.18182 

1.53741 

0.50616 

10 

5 

11 

13 

5 

4.81818 

2.89200 

0.57496 

10 

0 

11 

13 

6 

8.90909 

1.57826 

0.46355 

10 

5 

11 

13 

7 

8.18182 

1.94001 

0.51426 

10 

5 

11 

13 

8 

8.63636 

1.96330 

0.48956 

10 

5 

11 

13 

9 

9.00000 

1.48324 

0.43598 

10 

5 

11 

13 

10 

8.09091 

2.21154 

0.51426 

10 

5 

11 

13 

11 

7.18182 

1.88776 

0.51426 

10 

5 

11 

Weighted  Combined  Service  Class  Score  for  Class  13: 

Score: 8. 10540  Sigma: 1.02608  Rho: 0.82190  evaluators: 11. 00000 


for  Class  14  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

14 

1 

5.88889 

4.56740 

0.62854 

10 

0 

9 

14 

2 

5.55556 

3.71184 

0.64788 

10 

0 

9 

14 

3 

4.33333 

4.09268 

0.72008 

10 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  14: 
Score:5. 32793  Sigma: 2 . 53136  Rho:0. 87976  evaluators : 9 . 00000 


for  Class  15  Mean  scores  for  each  Criterion  are: 


Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

15 

1 

8.53846 

2.14536 

0.46154 

10 

5 

13 

15 

2 

8.84615 

1.95133 

0.49255 

10 

5 

13 

15 

3 

5.46154 

3.35697 

0.54393 

10 

0 

13 

15 

4 

2.07692 

3.20056 

0.54393 

10 

0 

13 

15 

5 

1.69231 

3.11942 

0.54393 

10 

0 

13 

15 

6 

9.15385 

1.86396 

0.46154 

10 

5 

13 

15 

7 

9.30769 

1.54837 

0.47419 

10 

5 

13 

Weighted  Combined  Service  Class  Score  for  Class  15: 

Score: 6. 86413  Sigma: 1.23743  Rho: 0.85022  evaluators: 13. 00000 
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CANDIDATE:  POSIX 


for 

Class 

16  Mean  scores  for 

each  Criterion  are: 

Cl 

Cr 

Score 

Std  dev 

Rho 

Max 

Min 

#Evaluators 

16 

1 

7.88889 

3.44400 

0.59835 

10 

0 

9 

16 

2 

7.88889 

3.44400 

0.59835 

10 

0 

9 

16 

3 

6.11111 

3.98260 

0.64788 

10 

0 

9 

16 

4 

6.11111 

3.98260 

0.59835 

10 

0 

9 

16 

5 

7.33333 

3.46410 

0.59835 

10 

0 

9 

16 

6 

3.55556 

4.18662 

0.62854 

10 

0 

9 

16 

7 

6.11111 

4.22624 

0.64788 

10 

0 

9 

16 

8 

6.11111 

4.22624 

0.64788 

10 

0 

9 

16 

9 

6.66667 

3.77492 

0.56656 

10 

0 

9 

16 

10 

7.00000 

3.50000 

0.59835 

10 

0 

9 

16 

11 

7.44444 

3.46811 

0.56656 

10 

0 

9 

16 

12 

7.33333 

3.96863 

0.56656 

10 

0 

9 

16 

13 

7.22222 

2.72845 

0.56656 

10 

3 

9 

16 

14 

7.00000 

2.78388 

0.59835 

10 

3 

9 

16 

15 

7.11111 

2.20479 

0.59835 

10 

5 

9 

16 

16 

7.77778 

3.41971 

0.59835 

10 

0 

9 

16 

17 

6.33333 

3.96863 

0.59835 

10 

0 

9 

16 

18 

5.77778 

3.73423 

0.59835 

10 

0 

9 

16 

19 

7.44444 

2.60342 

0.59835 

10 

3 

9 

16 

20 

7.33333 

3.24037 

0.59835 

10 

0 

9 

Weighted  Combined  Service  Class  Score  for  Class  16: 

Score:6. 84962  Sigma: 1.01341  Rho: 0.63187  evaluators: 9. 00000 
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APPENDIX  F 

ESSENTIAL  CRITERIA  PARTITION 


The  partition  of  requirements  by  DOP  requirement  area  is  shown  in  Table 
F-l.  In  the  table,  only  the  criteria  believed  to  be  "essential"  to  a  given 
system  aspect  are  listed.  The  headings  for  the  allocation  correspond  to  the 
high-level  requirements  as  follows:  R  «■  real-time;  D  ••  distributed;  H  - 
heterogeneity;  A  -  Ada;  S  -  security;  F  -  reliability  (fault  tolerance).  If  an 
entry  is  blank,  then  the  criterion  listed  is  not  considered  to  have  any 
special  significance  for  that  high-level  requirement.  If  the  entry  is  "E",  the 
criterion  is  considered  to  be  essential;  if  the  entry  is  "NE",  the  criterion 
is  considered  to  be  important  for  the  high-level  requirement,  but  not  so 
important  that  its  absence  should  be  considered  a  significant  black-mark 
against  a  candidate. 
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TABLE  F-l .  ESSENTIAL  CRITERIA  PARTITIONS 


CRITERION  ALLOCATION 


R 

D 

H 

A 

S 

F 

1.4 

Arch . Indep . 

E 

E 

1.10 

Ada  Binding 

NE 

NE 

NE 

1.11 

Other  Lang.  Bind. 

E 

E 

1.20 

Reaction  to  Blocking  E 

E 

NE 

E 

1.21 

Bounded  . . .  Times  E 

E 

NE 

E 

1.23 

Transaction  Sched.  E 

E 

NE 

1.24 

Access  Control 

E 

E 

E 

1.25 

Transparency 

E 

E 

1.26 

Resilience 

E 

E 

1.27 

Network  Partition 

E 

E 

E 

1.28 

Reference 

E 

E 

1.29 

Reallocation 

E 

E 

2.1 

Non-NGCR  Intf. 

NE 

3.1 

Audit  Data  Stor. 

E 

3.2 

Audit  Generation 

E 

3.3 

Audit  Record  Contents 

E 

3.4 

Audit  Data  Manip, 

E 

3.5 

Device  Labels 

E 

3.6 

Basic  DAC 

E 

3.7 

DAC  Incl./Exclu. 

E 

3.8 

DAC  Propagation 

E 

3.9 

Labelling  Exp.  Chan. 

E 

3.10 

Setting  Comm.  Labels 

E 

3.11 

Ident.  and  Authentic. 

E 

3.12 

Labelling  of  HR  Output 

E 

3.13 

Subj./Obj.  Labelling 

E 

3.14 

Label  Contents 

E 

3.15 

MAC  Policy 

E 

3.16 

MAC  Manip. 

E 

3.17 

Object  Reuse 

E 

3.18 

User  Notif.  of  Sens. 

E 

3.19 

Sens.  Label  Query 

E 

3.20 

System  Integrity 

E 

3.21 

Ident.  of  Users  by  Roles 

E 

3.22 

Least  Privilege 

E 

3.23 

Trusted  Path 

E 

3.24 

Trusted  Recovery 

E 

4.1 

Data  Interch.  Services 

NE 

NE 
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TABLE  F-l  (Cont.) 


CRITERION  ALLOCATION 

R  D  H  A  S  F 


5.1 

Event/Error  Receipt 

E 

E 

E 

5.2 

"  Distribution 

NE 

E 

E 

5.3 

"  Management 

E 

E 

E 

5.4 

Event  Logging 

NE 

E 

E  E 

5.5 

Enable/Disable.  Intrpts. 

E 

E 

E 

5.6 

Mash/Unmask  Intrpts . 

E 

E 

E 

6.1 

Contiguous  Read  of  File 

E 

NE 

6.2 

Protect  Area  in  File 

E 

NE 

6.3 

File  Mgt.  Sched. 

E 

E 

6.4 

File  Mgt.  Susp/Res  -  Proc 

E 

E 

6.5 

File  Mgt.  Block  Rqsts 

E 

E 

6.6 

Round  Robin  File  Mgt. 

NE 

NE 

6.7 

Open  a  File 

E 

E 

6.8 

Point  Within  File 

E 

E 

6.9 

Read  File 

E 

E 

6.10 

Close  File 

E 

E 

6.11 

Delete  File 

E 

E 

6.12 

Create  Directory 

NE 

E 

6.13 

Specify  Default  Dir. 

NE 

E 

6.14 

Delete  Directory 

NE 

E 

6.15 

Shadow  Files 

E 

E 

6.16 

Create  File 

E 

E 

6.17 

Query  File  Attrib. 

E 

E 

6.18 

Modify  File  Attrib. 

E 

E 

6.19 

Write  File 

E 

E 

6.20 

Write  Contig.  File 

E 

NE 

8.1 

Intf/Ctrl  of  F+/SN/1553 

NE 

NE 

8.2 

Intf/Ctrl  of  Others 

E 

NE 

8.3 

Reliable  Virt.  Circ.  Comm 

E 

E 

8.4 

Unrel.  Virt.  Circ.  Comm. 

E 

E 

8.5 

Rel.  Datagram  Transfer 

E 

E 

8.6 

Unrel.  Datagram  Transfer 

E 

E 

8.7 

Request-Reply  Service 

E 

E 

8.8 

Unrel.  Broadcst/Multicst 

E 

E 

8.9 

Rel.  Broadcst/Multicst 

E 

E 

8.10 

Atomic  Broadcst/Multicst 

E 

E 
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TABLE  F-l  (Cont.) 


CRITERION  ALLOCATION 

R  D  H  A  S  F 


9.1 

Create  Process 

E 

9.2 

Terminate  Process 

E 

9.3 

Start  Process 

E 

9.4 

Stop  Process 

E 

9.5 

Suspend  Process 

E 

9.6 

Resume  Process 

E 

9.7 

Delay  Process 

E 

9.8 

Interprocess  Comm. 

E 

9.9 

Exam.  Process  Attrib. 

NE 

9.10 

Modify  Process  Attrib. 

NE 

9.11 

Exam.  Process  Status 

E 

9.12 

Process  Id. 

E 

9.13 

Save/Restart  Process 

NE 

9.14 

Prog.  Mgt. 

NE 

10.1 

Debug  Support 

10.2 

Execution  History 

11.1 

Fault  Info.  Coll. 

NE 

11.2 

Fault  Info.  Request 

NE 

11.3 

Diag.  Test  Request 

E 

11.4 

Diag.  Test  Results 

E 

11.5 

Operational  Status 

E 

11.6 

Fault  Detec.  Threshhld. 

E 

11.7 

Fault  Isolation 

E 

11.8 

Fault  Response 

E 

11.9 

Reconfiguration 

E 

11.10 

En./Dis.  Sys .  Component 

E 

11.11 

Perf.  Mon. 

E 

11.12 

Set  Resource  Util.  Lims. 

E 

11.13 

Resource  Util.  Lim.  Viol. 

E 

11.14 

Chkpt.  Data  Structures 

12.2 

Virtual  Space  Locking 

E 

12.3 

Dyn.  Mem.  Alloc/Dealloc . 

NE 

12.5 

Shared  Memory 

E 

12.6 

Alloc/Dealloc/Mnt/Dsmnt 

12.7 

Designate  Control 

12.8 

Release  Control 

12.9 

Allocate  Resource 

12.10 

Dealloc.  Resource 

12.11 

Sys.  Resrc.  Rqts.  Spec. 

12.12 

Sys.  Resrc.  Capacity 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 


E  E 

E 

NE 

E  EE 

NE  E 

E  E 

E  NE 

E  NE 

E  EE 

E  E 

E  E 

E  E 

E  EE 

E  EE 

E  E 

E  E 

E  E 

NE  E 

NE 

E  NE 

E  NE 

E  NE 

E  NE 

E  NE 

E  NE 

E  NE 
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TABLE  F-l  (Cone.) 

CRITERION  ALLOCATION 

R  D  H  A  S  F 


13.1  Process  Synch.  E 

13.2  Mutual  Exclus.  E 

13.3  Cum.  Process  Execu.  Time  E 

13.4  Attach  Proc.  to  Event  NE 

13.5  Transac.  Sched.  Info.  NE 

13.6  Sched.  Delay  E 

13.7  Periodic  Sched.  E 

13.8  Mult.  Sched.  Policies  E 

13.9  Selec.  of  Sched.  Policy  E 

13.10  Mod.  of  Sched  Params.  E 

13.11  Precise  Sched.  E 

14.1  Image  Load 

14.2  Sys.  Init/Reinit 

14 . 3  Shutdown 

15.1  Read  Sel.  Clock  E 

15.2  Set  Sel.  Clock  E 

15.3  Synch  of  Sel.  Clocks  E 

15.4  Sel.  Prim.  Ref.  Clock  E 

15.5  Locate  Prim.  Ref.  Clock  E 

15.6  Timer  Services  E 

15.7  Precision  Clock  E 


16.1  Create  Task 

16.2  Abort  Task 

16.3  Suspend  Task 

16.4  Resume  Task 

16.5  Terminate  Task 

16.6  Restart  Task 

16.7  Ada  Task  Entry  Calls 

16.8  Ada  Task  Call  Accept/Sel. 

16.9  Access  Task  Chars. 

16.10  Mon.  Task  Execu.  Status 

16.11  Access  to  R-T  Clock 

16.12  Access  to  T-o-D  Clock 

16.13  Dyn.  Task  Priorities 

16.14  Sched.  Policy  Selection 

16.15  Mem.  Alloc/Dealloc . 

16.16  Interrupt  Binding 

16.17  En./Dis.  Intrpts. 

16.18  Mask/Unmask  Intrpts. 

16.19  Raise  Exception 

16.20  Ada  I/O  Support 


E 
E 

E 
E 

E 
E 
E 
E 

NE 

NE  ' 

E 
E 
E 
E 
E 
E 
E 
E 
E 

E 
E 
E 
E 
E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 


F-5 


m  w  m 


NAVSWC  TR  90-246 


DISTRIBUTION 


Copies 

Defense  Technical  Information  Center 
Cameron  Station 

Alexandria,  VA  22304-65145  12 

Library  of  Congress 

Attn:  Gift  and  Exchange  Division 

Washington,  DC  20540  4 

Center  for  Naval  Analyses 
4401  Hunt  Avenue 
P.0.  Box  16268 

Alexandria,  VA  22302-0268  2 

Assistant  Secretary  of  the  Navy 
(Research,  Engineering  and  Systems) 
Navy  Department 

Washington,  DC  20350-1000  2 

Assistant  Secretary  of  the  Navy 
(Shipbuilding  and  Logistics) 

Navy  Department 

Washington,  DC  20350-1000  2 

Chairman 

Naval  Postgraduate  School 
Attn:  Computer  Science  Department  1 
Administrative  Sciences 


Department 

Engineering  Department 
Library  Department 
Monterey,  CA  93943 

Chief  of  Naval  Education  and 
Training  Center 
Naval  Air  Station 
Pensacola,  FL  32508 

Chief  of  Naval  Operations 
Navy  Department 
Attn:  Codes  OP-04 
OP-22 
OP-35 
OP-55 
OP- 94 
OP-95 
OP-98 


1 

1 

1 

1 

1 

1 

1 


Washington, 


OP-112 
OP-211 
OP-224 
OP-351 
OP-551 
OP -940 
OP- 941 
OP- 942 
OP- 944 
OP- 945 
OP-982 
OP-983 

DC  20350-2000 


Conies 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 


Chief  of  Naval  Research 
800  North  Quincy  St. 

Arlington,  VA  22217-5000  1 

Commandant 
US  Coast  Guard 

Attn:  Codes  G-FQA  1 

G-TES 

(LCRD  Rex  Buddenberg)  1 
2100  2nd  St.  SW 
Washington,  DC  20593 

Commander  3 

Technical  Director  1 

David  Taylor  Research  Center 
Bethesda,  MD  20084-5000 

Commander  1 

Technical  Director  1 

Naval  Air  Development  Center 
Warminster,  PA  18974-5000 

Commander  1 

Technical  Director  1 

Naval  Air  Systems  Command 
Attn:  Code  AIR-5466  (Barry  Corson) 
Washington,  DC  20361 

Commander  30 

Technical  Director  1 

Naval  Air  Systems  Command 
Washington,  DC  20361-0001 


(1) 


NAVSWC  TR  90-246 


DISTRIBUTION  (Cont.) 


Copies 


Commander  1 

Technical  Director  1 

Naval  Air  Test  Center 
Patuxent  River,  MD  20670-5304 

Commander  1 

Technical  Director  1 

Naval  Coastal  Systems  Center 
Panama  City,  FL  32407-5000 

Commander  1 

Technical  Director  1 


Naval  Data  Automation  Command 
Washington  Navy  Yard 
Washington,  DC  20374-1662 

Commander  1 

Technical  Director  1 

Naval  Data  Automation  Command 
Washington  Navy  Yard 
Attn:  Code  14  (Paul  Robinson) 
Washington,  DC  20374-1662 


Commander  1 

Technical  Director  1 

Naval  Education  and  Training  Center 
Newport,  RI  02841-5000 

Commander  1 

Technical  Director  1 


Naval  Intelligence  Command 
4600  Silver  Hill  Road 
Washington,  DC  20389-5000 


Commander  1 

Technical  Director  1 

Naval  Ocean  Systems  Center 
San  Diego,  CA  92152-5000 

Commander  1 

Technical  Director  1 

Naval  Research  laboratory 
Washington,  DC  20375 

Commander  30 

Technical  Director  1 

Naval  Sea  Systems  Command 
Washington,  DC  20362-5101 


Copies 


Commander  1 

Technical  Director  1 

Naval  Sea  Systems  Command 
Attn:  Codes  SEA- 90  1 

SEA-06D1  1 

PMS-412  1 

Washington,  DC  20362-5101 

Commander  1 

Technical  Director  1 


Naval  Security  Group  Command 
Naval  Security  Group  Headquarters 
3801  Nebraska  Avenue,  NW 
Washington,  DC  20393-5210 


Commander  1 

Technical  Director  1 

Naval  Supply  Systems  Command 
Washington,  DC  20376-5000 

Commander  1 

Technical  Director  1 


Naval  Telecommunications  Command 
4401  Massachusetts  Ave,  NW 
Washington,  DC  20394-5290 


Commander  1 

Technical  Director  1 

Naval  Underwater  Systems  Center 
Newport,  RI  02841-5047 

Commander  1 

Technical  Director  1 

Naval  Weapons  Center 
China  Lake,  CA  93555-6001 

Commander  1 

Technical  Director  1 

Operational  Test  and 
Evaluation  Force 
Norfolk,  VA  23511-6388 


Commander  1 

Technical  Director  1 

Pacific  Missile  Test  Center 
Point  Mugu,  CA  93042-5000 


(2) 


NAVSWC  TR  90-246 


DISTRIBUTION  (Cont.) 


Copies 

Commanding  Officer 
Fleet  Combat  Direction  Systems 
Support  Activity 
Dam  Neck 

Virginia  Beach,  VA  23461-5300  1 

Commanding  Officer 

Fleet  Combat  Direction  Systems 

Support  Activity 

San  Diego,  CA  92147-5081  1 

Commanding  Officer 
Integrated  Combat  Systems 
Test  Facility 

San  Diego,  CA  92152-6900  1 

Commanding  Officer 

Naval  Air  Engineering  Center 

Attn:  Code  9322  1 

Lakehurst,  NJ  08733-5100 

Commanding  Officer 

Naval  Avionics  Center 

6000  East  21st  Street 

Indianapolis,  IN  46219-2189  1 

Commanding  Officer 

Navy  Data  Automation  Facility 

Box  140 

Pearl  Harbor,  HI  96860  1 

Commanding  Officer 

Navy  Data  Automation  Facility 

Bldg  1A 

Newport,  RI  02841  1 

Commanding  Officer 
Naval  Education  and  Training 
Security  Assistance  Field  Activity 
Pensacola,  FL  32508-5100  1 

Commanding  Officer 
Naval  Electonic  Systems 
Engineering  Center 
P.0.  Box  80337 

San  Diego,  CA  92138-3288  1 

Commanding  Officer 
Naval  Electronic  Systems 
Engineering  Center 

Vallejo,  CA  94592-5017  1 


Conies 

Commanding  Officer 
Naval  Electronic  Systems 
Engineering  Center 
P.O.  Box  55 

Portsmouth,  VA  23705-0055  1 

Commanding  Officer 

Naval  Electronic  Systems 

Security  Engineering  Center 

Naval  Security  Station 

3801  Nebraska  Avenue,  NW 

Washington,  DC  20393-5270  1 

Commanding  Officer 

Naval  Electronic  Systems 

Engineering  Center 

4600  Marriott  Drive 

North  Charleston,  SC  29418-6504  1 

Commanding  Officer 

Naval  Electronic  Systems 

Engineering  Activity 

St.  Inigoes,  MD  20684-0010  1 

Commanding  Officer 
Navy  Management  Systems 
Support  Office 

Norfolk,  VA  23511-6694  1 

Commanding  Officer 

Naval  Ordnance  Station 

Louisville,  KY  40214  1 

Commanding  Officer 

Naval  Ordnance  Station 

Attn:  Code  524  1 

Indian  Head,  MD  20640-5000 

Commanding  Officer 
Navy  Regional  Data 
Automation  Center 
Naval  Air  Station 

Alameda,  CA  94501-5007  1 

Commanding  Officer 
Navy  Regional  Data 
Automation  Center 
Naval  Air  Station 

San  Diego,  CA  92135  1 


(3) 


NAVSWC  TR  90-246 
DISTRIBUTION  (Cont.) 


Copies 

Commanding  Officer 
Navy  Regional  D^ta 
Automation  Center 
Naval  Air  Station 

Jacksonville,  FL  32212-0111  1 

Commanding  Officer 
Navy  Regional  Data 
Automation  Center 
Naval  Air  Station 

Pensacola,  FL  32508-6100  1 

Commanding  Officer 
Naval  Regional  Data 
Automation  Center 

New  Orleans,  LA  70145  1 

Commanding  Officer 
Navy  Regional  Data 
Automation  Center 

Norfolk,  VA  23511  1 

Commanding  Officer 

Naval  Ship  Weapon  Systems 

Engineering  Station 

Port  Hueneme,  CA  93043-5007  1 

Commanding  Officer 

Navy  Space  Systems  Activity 

P.O.  Box  92960 

Worldway  Postal  Center 

Los  Angeles,  CA  90009-2960  1 

Commanding  Officer 

Navy  Tactical  Interoperability 

Support  Activity 

San  Diego,  CA  92147  1 

Commanding  Officer 

Pacific  Missile  Test  Center 

Attn:  Code  4003  (Lucille  Cook)  1 

Pt.  Mugu,  CA  93042 

Fleet  Combat  Direction  System 
Attn:  Code  8d  1 

George  Robertson  1 

San  Diego,  CA  92147 

Navair  Air- 54664 

Attn:  Victor  Skullman  1 

Washington,  DC  20361 


Copies 

Naval  Air  Development  Center 

Attn:  Codes  7031  Tricia  Obemdorf  1 

7032  (Frank  Prindle)  1 

7033 

(Dr.  Carl  Schmiedekamp)l 
Warminster,  PA  18974-5000 

Naval  Air  Systems  Command 

Attn:  Code  PMA209e  (Bruce  Anderson)  1 

Washington,  DC  20361-0001 

Naval  Air  Test  Center 
Attn:  Codes  SY30  (Jerry  Abrams)  1 
SY30  (Luke  Campbell)  1 
Patuxent  River,  MD  20670 

Naval  Avionics  Center 
Attn:  John  S.  Johnson  1 

6000  East  21st  (Mail  Stop  825) 
Indianapolis,  IN  46219-2189 

Naval  Ocean  Systems  Center 

Attn:  Code  41  (Art  Justice)  1 

San  Diego,  Ca  92152-5000 

Naval  Ocean  Systems  Center 
Attn:  Codes  412  (Rich  Bergman)  1 
412  (Warren  Loper)  1 
412  (Dwight  Wilcox)  1 
855  (Will  Fitzgerald)  1 
855  (Roger  Hall)  1 

San  Diego,  CA  92152-5000 

Naval  Research  Laboratory 
Attn:  Codes  5540b  (H.  0.  Lubbes)  1 
5544  (Maria  Voreh)  1 
4555  Overlook  Ave.  SE 
Washington,  DC  20375-5000 

Naval  Underwater  System  Center 
Attn:  Codes  2211  (Gregory  Bussiere)l 
2211  (Gail  Holmes)  1 
2221  (Tom  Conrad)  1 
2221  (Jim  Oblinger)  1 
2221 

(Bonnie  Netten-Reilly)  1 
2221  (Dan  Juttelstad)  1 
2221 

(LCDR  Robert  Voigt)  1 
Newport,  RI  02841-5047 


(4) 


4 


■> 


V 


NAVSWC  TR  90-246 
DISTRIBUTION  (Cont.) 


Naval  Air  Systems  Command,  HQ 
Attn:  Matthew  Emerson  1 

Washington,  DC  20361-1014 

Naval  Weapons  Center 
Attn:  Codes  31c  (John  Zenor)  1 

3922 

(Lt.  K.  Fairbanks,  Jr.)l 
3922  (Carl  Hall)  1 

China  Lake,  CA  93555-6001 

Naval  Underwater  Systems  Command 
Attn:  Codes  2153 

(Rosemary  Howbrigg)  1 
3313  (John  Gregor)  1 

New  London,  CT  06320 

Office  of  Naval  Research 

Attn:  Code  1133 

(Andre  Van  Tilborg)  1 

800  North  Quincy  Street 
Washington,  DC  22217-5000 


Officer  in  Charge 
Annapolis  Laboratory 
David  Taylor  Research  Center 
Annapolis,  MD  21402-5067  1 


Officer  in  Charge 
Naval  Electronic  Systems 
Engineering  Center  Detachment 
P.O.  Box  286 
Naval  Station 

Mayport,  FL  32229  1 


Officer  in  Charge 

Naval  Electronic  Systems 

Engineering  Activity  Detachment 

Bldg  76,  Naval  Base 

Philadelphia,  PA  19112  1 


Officer  in  Charge 
Naval  Electronic  Systems 
Engineering  Center  Detachment 
P.O.  Box  286 
Naval  Station 

Mayport,  FL  32229  1 


Conies 

Officer  in  Charge 

Naval  Electronic  Systems  Detachment 


Naval  Air  Station 

Patuxent  River,  MD  20670  1 

Officer  in  Charge 

Naval  Electronic  Systems  Detachment 
Defense  Activities 

Mechanicsburg,  PA  17055  1 

Officer  in  Charge 

New  London  Laboratory 

Naval  Underwater  Systems  Center 

New  London,  CT  06320-5994  1 

Pentagon,  Room  5B731 
Attn:  ASN (RD&A) - D0NIRM 

(Ted  E.  Senator)  1 

Washington,  DC  20350-1000 

Phalanx  -  Naval  Ordnance  Station 
Attn:  Code  50  (John  G.  Shea)  1 

Louisville,  KY  40214-5001 


President 

Board  of  Inspection  and  Survey 
Navy  Department 

Washington,  DC  20372-5100  1 

Program  Manager 
Theater  Nuclear  Warfare 
Program  Office 
Department  of  the  Navy 
Rome  Air  Development  Center 
COTD 

Attn:  Tom  Lawrence  1 

Griffiss  AFB,  NY  13441-5700 

Senior  Member 

Board  of  Inspection  and 

Survey  Pacific 

Bldg  116,  Naval  Station 

San  Diego,  CA  92136-5107  1 

Senior  Member 

Board  of  Inspection  and 

Survey  Atlantic 

Norfolk,  VA  23520  1 


(5) 


NAVSWC  TR  90-246 


DISTRIBUTION  (Cont.) 


Copies 


Copies 


Space  &  Naval  Warfare  System  Command 


Attn:  Codes  00  1 

00E  1 

003  1 

003-22  1 

30  1 

32  1 

32  (RADM  George  Wagner)  1 
32A  1 

162-13A 

(Francis  Deckelman)  1 
321  1 

324  (Philip  Andrews)  1 
324  (Norma  Stopyra)  1 
324A  (CDR  Rick  Barbour)  1 
3212  1 

3212  (CDR  Martin  Romeo)  1 
3212S  1 


3243  (Alexander  Lewin)  1 
3243  (Hank  Mendenhall)  1 
3243b  (LCDR  Ron  Owens)  1 
3243b  (Patricia  Pell)  1 
32431b  (Kar  Yee  Chan)  1 


PD -40  1 
PD-50  1 
PD-60  1 
PD-70  1 
PD^70C  1 
PD-70-1  1 
PD-70-2  1 
PD-70-3  1 
PD-70-4  1 
PD-80  1 
PMW-141  1 
PMW-142  1 
PMW-14b  1 
PWM-144  1 
PMW-145  1 
PMW-146  1 
PMW-147  1 
PMW-151  1 
PMW-152  1 
PMW-153  1 
PMW-155  1 
PMW-156  1 
PMW-159  1 
PMW-161  1 
PMW-162  1 
PMW-163  1 
PMW-164  1 
PMW-174  1 


PMW-180 

PMW-181 

PMW-182 

PMW-183 

PMW-183-421 

(Richard  Cockerill) 

PMW-184 

PMW-610 

PMW-620 

PMW-630 

Washington,  DC  20363-5100 


1 

1 

1 

1 

1 

1 

1 

1 

1 


Commandant  of  the  Marine  Corps 
Headquarters,  US  Marine  Corps 


Attn:  Codes  C21  (Doug  Smith)  1 

CCA- 51  1 

LMA-1  1 

MCRDAC  1 

OCIER  1 

PSE  1 

Washington,  DC  20380-0001 


Commanding  General 
Marine  Corps  Research, 

Development  and  Acquisition  Command 
Attn:  Code  MCRDAC  1 

Quantico,  VA  22134 

Commanding  General 

Marine  Corps  Development 

and  Education  Command 

Quantico,  VA  22134  1 


Commanding  General 
Marine  Corps  Research, 

Development  and  Acquisition 
Command 

Attn:  Code  PSE  1 

Washington,  DC  20380-0001 

Commanding  Officer 

Marine  Corps  Tactical  Systems 

Support  Activity 

Camp  Pendleton,  CA  92055-5080  1 

United  States  Army  Information 
Systems  Command 
Attn:  ASQB-SEP-C 

Major  Terry  Fong 
Ft.  Huachuca,  AZ  85613 


1 

1 


NAVSWC  TR  90-246 


■< 


3 


t 


V 


DISTRIBUTION  (Cont.) 


Copies 


Copies 


Headquarters,  AFSC/ENR 
Andrews  AFB 

Attn:  Cpt.  Peter  M.  Vaccaro  1 

Washington,  DC  20334-5000 

USAF 

SSC/XPT  Gunter  AFB 

Attn:  Elizabeth  Crouse  1 

AL  36114 

USAF  (Office  of  the  Secretary) 
SAF/AQXA 

Attn:  Lt.  Col  Richard  Gross  1 

Washington,  DC  20330 

NASA  (Langley  Research  Center) 

Attn:  Sharon  0.  Beskenis  1 

Mail  Stop  478 
Hampton,  VA  23665-5225 

Cruise  Missiles  Project  And  Unmanned 
Aerial  Vehicles  Joint  Project 
Attn:  Dr.  C.  Thomas  1 

Bo.  F.  Tung  1 

Washington,  DC  20361-1014 

Defense  Advance  Research 
Project  Agency 

Information  Science  Technology  Office 
Attn:  Dr.  William  Scherlis  1 

1400  Wilson  Blvd 
Arlington,  VA  22209-2308 

Defense  Systems  Management  College 


Technical  Management  Division 

Ft.  Belvoir ,  VA  22060  1 

Director  -  Program  Office 

Joint  Cruise  Missiles 

Department  of  the  Navy 

Washington,  DC"  20363  1 

Director 

Strategic  Systems  Program  Office 
Washington,  DC  50376-5002  1 

Jet  Propulsion  Lab 

Attn:  Dr.  Frank  Mathur  1 

Dr.  John  Rohr  1 

4800  Oak  Grove  Drive 
Pasadena,  CA  91109 


National  Institude  of  Standard  and 


Technology 

Technology  Building,  Rm  B266 
Attn:  Gary  Fisher  1 

Jim  Hall  1 

Gaithersburg,  MD  20899 

Advanced  System  Technologies,  Inc. 
Attn:  Gary  J.  Wright  1 

5113  Leesburg  Pike,  Suite  514 
Falls  Church,  VA  22041 

Advanced  Technology  &  Research  Corp. 
Attn:  Riley  H.  Mayhall,  Jr.  1 

Adrien  J.  Meskin  1 

14900  Sweitzer  Lane 
Laurel,  MD  20707 


American  Systems  Corporation 
Attn:  Michael  Cornelison  1 

14200  Park  Meadow  Drive 
Chantilly,  VA  22021 

Arnold  Associates 

Attn:  Dr.  Charles  R.  Arnold  1 

P.0.  Box  1151 

New  London,  CT  06320-1151 

BBN  Systems  &  Technologies  Corp. 

Attn:  Ken  Schroeder  1 

Dr.  Stephen  T.  Vinter  1 

10  Moulton  Street 
Cambridge,  MA  02138 

Booz,  Allen  &  Hamilton  Inc. 

Attn:  Mark  S.  Karan  1 

Ed  Rodriquez  1 

4330  East  West  Hwy. 

Bethesda,  MD  20814 

CAE-Link  Flight  Simulation 

Attn:  L.W.  Burns  1 

2224  Bay  Area  Blvd. 

Houston,  TX  77058 

Carnegie  Mellon  University 
School  of  Computer  Science 

Attn:  Dr.  Hideyuki  Tokuda  1 

Pittsburgh,  PA  15213-3890 


(7) 


NAVSWC  TR  90-246 


DISTRIBUTION  (Cont.) 


Copies 

Carnegie  Mellon  University- 
Software  Engineering  Institute 
Attn:  Mark  Borger  1 

Pittsburgh,  PA  15213-7614 

CEA  Incorporated 

Attn:  Gail  Driskill  1 

1680  East  Gude  Drive 
Rockville,  MD  20850 

Clemson  University 

Attn:  Dr.  Thomas  Drake  1 

117  Riggs  Hall 
Clemson,  SC  29634-0915 

Clemson  University 
Dept  of  Electrical  & 

Computer  Engineering 

Attn:  James  Leathrum  1 

Clemson,  SC  29634 

Compunetics 

Attn:  Dr.  Giorgio  Coraluppi  1 

2000  Eldo  Rd 
Monroeville,  PA  15146 

Computer  Based  Systems,  Inc. 

(Suite  503b) 

Attn:  Carl  Reinert  1 

605  Louis  Drive 
Warminster,  PA  18974 

Computer  Sciences  Corp  (MC263) 

Attn:  P.C.  Wood  1 

2066  Leesburg  Pike 
Falls  Church,  VA  22041 

Computer  Sciences  Corporation 
Attn:  Barry  M.  Gladstone  1 

William  C.  Lev  1 

304  West  Route  38,  P.0.  Box  N 
Moorestown,  NJ  08053 

Concurrent  Computer  Corporation 
Attn:  Balakrishnan  Dasarathy  1 

E.  Douglas  Jensen  1 

1  Technology  Way 
Westford,  MA  01886 


Copies 

Control  Data  Corporation 
Attn:  Robert  J.  Cunius  1 

6003  Executive  Blvd 
Rockville,  MD  20852 

Control  Data  Corporation 

Attn:  Gary  Jaeger  1 

P.0.  Box  609  (M/S  HQG  511) 

Minneapolis,  MN  55425 

Cray  Research,  Inc. 

Attn:  Claire  Garrett  1 

National  Place,  1331  Pennsylvania  Ave. 
Washington,  DC  20004 

Data  General  Corporation  (Suite  200) 
Attn:  Lee  Trumbore  1 

7927  Jones  Branch  Drive 
McLean,  VA  22102 

Defense  Systems  Management  College 

Technical  Management  Division 

Ft.  Belvoir ,  VA  22060  1 

DGM  &  S 

Attn:  Steve  Davis  1 

1025  Briggs  Rd  (Suite  100) 

Mt.  Laurel,  MD  08054 

Digital  Equipment  Corporation 
Attn:  John  F.  Reed  1 

1  Continental  Blvd 
Merrimack,  NH  03054 

DY-4  Systems  Inc. 

Attn:  Jeremy  James  1 

21  Credit  Union  Way 
Napean,  Ontario 

DY-4  Systems  Inc. 

Attn:  Kim  Clohessy  1 

21  Fitzgerald  Rd 
Napean,  Ontario 

Dynamics  Research  Corporation 
Attn:  Ray  Gretlein  1 

11545  West  Bernardo  Ct  Suite  100 
San  Diego,  CA  92127 


NAVSWC  TR  90-246 


DISTRIBUTION  (Cont.) 


Copies 


Copies 


Dynamics  Research  Corporation 
Attn:  Geoffrey  Lampshire  1 

60  Frontage  Road 
Andover,  MA  01810 

Dynamics  Research  Corporation 
Attn:  Philip  L.  Palatt  1 

1755  Jefferson  Davis  Hwy  y/802 
Arlington,  VA  22202 

Electronic  Data  Systems 

Attn:  Norma  Lesher  1 

13600  Eds  Drive 

Herndon,  VA  22071 

ESL 

Attn:  Gordon  Caswell  1 

8706  Victoria  Road 
Springfield,  VA  22151 

GE  Advanced  Technology  Labs 
Attn:  Dr.  John  F.  Nixon  1 

GE  Corporate  Research  and  Development 
Attn:  Walter  Dixon  1 

P.0.  Box  8 

Schenectady,  NY  12301 

GE  Ocean  Systems  Division 
Attn:  Howard  Hollander  1 

P.0.  Box  4840 
Syracuse,  NY  13221-4840 

Hewlett  Packard  (Suite  1100) 

Attn:  Louis  Reveron  1 

900  17th  Street,  NW 
Washington,  DC  20006 

Honeywell  Fed  System  (MS  1010) 

Attn:  Lester  Fraim  1 

7900  Westpark  Drive 
Mclean,  VA  22012 

Honeywell,  Inc,  (MN65-2100) 

Attn:  Dean  Knudson  1 

3660  Technology  Drive 
Minneapolis,  MN  55418 

IBM  Corporation  409 

Attn:  Dr.  Douglass  Locke  1 

6600  Rockledge  Drive 

Bethesda,  MD  20817 


IBM-FSD  MD-0210 
Attn:  Dave  Vogel 
Route  17c 
Owego,  NY  13827 

IBM  Manassas 
Bldg.  101/084 
Attn:  Dale  Carrow 
Pat  Watson 
9500  Godwin  Drive 
Manassas,  VA  22110 

Industrial  Programming,  Inc. 
Attn:  Carol  Sigda 
100  Jericho  Quadrangle 
Jericho,  NY  11753 

Institute  For  Defense  Analyses 
Attn:  Dr.  Karen  Gordon 
1801  North  Beauregard  St 
C&SE  DIV 

Alexandria,  VA  22311 

Integrated  Software  Inc. 

Attn:  Sam  Harbaugh 
1945  Palm  Bay  Rd  #7 
Palm  Bay,  FL  32905 

Intel 

Attn:  Richard  Dvorchak 
Tim  Saponas 

5200  NE  Elam  Young  Pkwy 
Hillsboro,  OR  97124 

John  Hopkins  University 
Applied  Physics  Lab 
Attn:  Eric  Conn 
Bill  Shaw 
Laurel,  MD  20707 

Johns  Hopkins  University 
Attn:  Dr.  Mars  J.  Gralia 
Johns  Hopkins  Road 
Laurel ,  MD  20723 

Johns  Hopkins  University 
Computer  Science  Dept. 

Attn:  Dr.  Gerald  Masson 
Baltimore,  MD  21210 


(9) 


NAVbWC  iK  SU-/4D 


DISTRIBUTION  (Cont.) 


Copies 

Litton  Data  Systems 

Attn:  A1  Ruemke  1 

1755  Jefferson  Davis  Hwy 
Suite  510,  CS  5 
Arlington,  VA  22202 

Litton  Data  Systems 
Attn:  Neil  Henderson  1 

2810  Old  Mobile  Hwy  Dept  910 
Pascagoula,  MS  39567 

Logicon,  Inc. 

Attn:  Barry  C.  Stauffer  1 

2100  Washington  Blvd 
Arlington,  VA  22204 

Martin  Marietta  Energy  Systems  Inc. 
Attn:  Mike  R.  Cossey  1 

P.0.  Box  2003,  K-1037 ,  M/S-7346 
Oak  Ridge,  TN  37831-7346 

Mitre 

Attn:  Antonio  Carangelo  1 

Burlington  Road 
MS  B325 

Bedford,  MA  01730 
Mnenomics  Inc. 

Attn:  R.T.  Semraes  1 

6304  Potomac  Ave 
Alexandria,  VA  22307 

Motorola 

Attn:  Richard  Vanderlin  1 

2900  South  Diablo  Way 

MD212 

Tempe,  AZ  85282 

Odyssey  Research  Associates,  Inc. 

Attn:  Douglas  Weber  1 

301A  Harris  B.  Dates  Drive 
Ithaca,  NY  14850-1313 

Offer  Pazy 

3702  Stearns  Hill  Rd. 

Waltham,  MA  02154  1 

Open  Software  Foundation  (Suite  200) 
Attn:  William  H.  Finley  1 

620  Herndon  Parkway 
Herndon,  VA  22076 


Copies 

Oracle  Corporation 
Attn:  Bob  Kowalski 

Dennis  J.  Reilly 
3  Bethesda  Metro  Ctr  Suite 
Bethesda,  MD  20814 

PICHTR 

Attn:  Michael  L.  Morgan  1 

711  Kapiolani  Blvd 
Honolulu,  HI  96813 

Ratheon  Company 

Attn:  Lawrence  E.  Rainville  1 

1001  Boston  Post  Road 
M/A  1-1-1475 
Marlboro,  MA  01752 

Raytheon  Company  (M/S98A) 

Attn:  Isobel  Davis  1 

Joseph  Gwinn  1 

528  Boston  Post  Road 
Sudbury,  MA  01776 

Ready  Systems 

Attn:  Shellie  Gascon  1 

7855  Walker  Drive,  Suite  420 
Greenbelt,  MD  20770 

Ready  Systems  Inc 

Attn:  James  F.  Ready  1 

Dalibor  F.  Vrsalovic  1 

470  Potrero  Ave. 

Sunnyvale,  CA  94086 

Rockwell  International 

Attn:  Larry  Daubert  1 

3370  Miraloma  MS-BB12 

Anaheim,  CA  92803 

Rockwell  International 
Attn:  Steve  Bishop  1 

Calvin  Dodd  1 

Frank  Martin  1 

1745  Jefferson  Davis  Highway 
Arlington,  VA  22202 

SCTC  Inc. 

Attn:  Barry  Miracle  1 

1210  West  County  Rd.  E  Suite  100 
Arden  Hills ,  MN  55112 


1 

1 


1400 


(10) 


NAVSWC  TR  90-246 


DISTRIBUTION  (Cont.) 


Copies 

Softech 

Attn:  Bill  Buseman  1 

2000  North  Beauregard  Street 
Alexandria,  VA  22311-1794 

Syscon  Corporation 

Attn:  Robert  F.  Owens  1 

1000  Thomas  Jefferson  Street,  NW 
Washington,  DC  20007 

Systems  Exploration,  Inc. 

Attn:  Bob  Welty  1 

2121  Crystal  Park  Drive,  if 514 
Arlington,  VA  22202 

Teledyne  Brown  Engineering  (Suite  200) 
Attn:  Edward  C.  Rozelle  1 

3700  Pender  Drive 
Fairfax,  VA  22030 

Telesoft 

Attn:  Ken  Dixon  1 

Rob  Hadley  1 

5959  Cornerstone  Court  West 
San  Diego,  CA  92121-9891 

Texas  Instruments 

Attn:  Karen  S.  Johnson  1 

P.0.  Box  650311,  M/S  3986 
Dallas,  TX  75265 

Texas  Instruments 

Attn:  John  C.  Jensen  1 

P.0.  Box  655474  (MS  238) 

Dallas,  TX  75265 

The  Numerix  Federal  Systems  Corp. 

Attn:  Donald  Olson  1 

8260  Greensboro  Drive,  Suite  135 
Mclean,  VA  22102 

Trusted  Information  Systems 
Attn:  Homayoon  Taj  alii  1 

3060  Washington  Road 
Glenwood,  MD  21738 

TRW  Systems  Division 
Attn:  Steven  H.  Barry  1 

Ann  Marmor  Squires  1 

2751  Prosperity  Avenue 
Fairfax,  VA  22031 


Conies 

UC  Irvine 

Electrical  &  Comp  Engineering 
Attn:  Prof.  Kane  Kim  1 

Room  544  E 
Irvine,  CA  92717 

UCLA  (Computer  Science  Department) 
Attn:  Wesley  Chu,  Chair  1 

3713  Boelter  Hall 
Los  Angeles,  CA  90024 

Unisys 

Attn:  James  Lonjers  1 

70  Swedesford  Rd 
P.0.  Box  517 
Paoli,  PA  19301 


Unisys  Computer  Systems 

Attn:  Barbara  Haleen  1 

Mike  Karnrad  1 

Kari  K.  Kruempel  1 

Dr.  Del  Swanson  1 

P.0.  Box  64525 
St.  Paul,  MN  55164-0525 


Unisys  Corporation 

Attn:  Hans  W.  Polzer  1 

12010  Sunrise  Valley  Drive 
Reston,  VA  22090 

University  of  Pennsylvania 
Dept  of  Computer  Science 


Attn:  Insup  Lee  1 

Noah  Prywes  1 

Philadelphia,  PA  19104 

Vitro  Corporation 

Attn:  Charles  Brown  1 

Walter  Greenspon  1 

Donald  Sassaman  1 

14000  Georgia  Avenue 
Silver  Spring,  MD  20906 


Vitro  Corporation 

Attn:  Tim  Darby  1 

Bldg  A- 336,  15  West  Gude  Drive 
Rockville,  MD  20850 


NAVSWC  TK  yu-^4b 


DISTRIBUTION  (Cont.) 


Copies 


Copies 


Westinghouse  Electric  Corp. 


Attn:  Thomas  T.  Wishart  1 

P.0.  Box  746 
Baltimore,  MD  21203 

Internal  Distribution: 

E231  2 

E232  3 

N32  (R.  Reyes)  1 

N305  (D.  Green)  1 

U042  (J.  Bilmanis)  1 

U30  (A.  Jox)  1 

U30  (P.  Hwang)  1 

U33  (S.  Howell)  10 

U33  (T.  Park)  1 

U33  (H.  Roth)  1 

U33  (C.  Sadek)  1 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  lor  thu  collection  ot  information  n  estimated  to  average  t  hour  per  response,  including  tfte  time  tor  reviewing  instructions,  searching  eusting  data 
sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information  Send  comments  regarding  this  burden  estimate  or  any  other 
aspen  of  this  collection  of  information  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services.  Directorate  for  Information  Operations  and 
Reports,  I?  1 S  Jefferson  Davis  Highway.  Suite  1204.  Arlington.  VA  22202-4302.  and  to  the  Office  of  Management  and  Budget.  Paperwork  Redunion  Project  (0704  0 188). 
Washington.  DC  20S03 


1.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE 

7  Mav  1990 


3.  REPORT  TYPE  AND  DATES  COVERED 


4.  TITLE  AND  SUBTITLE 

Evaluation  Results  Report  for  Next  Generation  Computer  Resources  Operating 

Systems  Interface  Baseline  Selection 

5.  FUNDING  NUMBERS 

6.  AUTHOR(S) 

Steven  L.  Howell,  Editor 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Surface  Warfare  Center  (U33) 

10901  New  Hampshire  Avenue 

Silver  Spring,  MD  20903-5000 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

NAVSWC  TR  90-246 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Space  and  Naval  Warfare  Systems  Command  (SPAWAR  3243) 
Washington,  DC  20365-5109 

10.  SPONSORING/MONITORING 

AGENCY  REPORT  NUMBER 

11.  SUPPLEMENTARY  NOTES 

12a.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

12b.  DISTRIBUTION  CODE 

13.  ABSTRACT  (Maximum  200  wordi) 

This  report  summarizes  the  results  of  the  Next  Generation  Computer  Resources  (NGCR)  Operating 
Systems  Standards  Working  Group  (OSSWG)  evaluation  of  candidates  for  the  Operating  System  Interface 
(OSIP)  Baseline. 


14.  SUBJECT  TERMS 

NGCR  OSSWG 

15.  NUMBER  OF  PAGES 

170 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION 
OF  REPORT 

UNCLASSIFIED 

18.  SECURITY  CLASSIFICATION 
OF  THIS  PAGE 

UNCLASSIFIED 

19.  SECURITY  CLASSIFICATION 
OF  ABSTRACT 

UNCLASSIFIED 

20.  LIMITATION  OF 

ABSTRACT 

SAR 

